[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:00 dist-test-slave-pgkr krb5kdc[20387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:00 dist-test-slave-pgkr krb5kdc[20387](info): set up 2 sockets
Aug 08 02:24:00 dist-test-slave-pgkr krb5kdc[20387](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 02:24:01.108773 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:01 dist-test-slave-pgkr krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:01.310978 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40017
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:40017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:01.865456 20400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:01.865882 20400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:01.866140 20400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:01.875692 20400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:01.875919 20400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:01.876078 20400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:01.876233 20400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:01.876363 20400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:01.889068 20400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:40017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:01.890694 20400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:01.892253 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:01.908588 20405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:01.908690 20406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:01.908788 20408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:01.912250 20400 server_base.cc:1034] running on GCE node
I20260808 02:24:01.913276 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:01.915117 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:01.916445 20400 hybrid_clock.cc:648] HybridClock initialized: now 1786155841916345 us; error 75 us; skew 500 ppm
I20260808 02:24:01.916978 20400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:01 dist-test-slave-pgkr krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155841, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:01.921877 20400 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:01.924198 20400 webserver.cc:460] Webserver started at http://127.19.231.62:37837/ using document root <none> and password file <none>
I20260808 02:24:01.925153 20400 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:01.925365 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:01.925781 20400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:01.929685 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "837f3578f0414e62b6fede451ebd3d91"
format_stamp: "Formatted at 2026-08-08 02:24:01 on dist-test-slave-pgkr"
server_key: "9fa34ee9564d0f4d0ca8c8dcff977c7d"
server_key_iv: "4cfdd1ee53cef46c7493c9f4d1158ef1"
server_key_version: "encryptionkey@0"
I20260808 02:24:01.930670 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "837f3578f0414e62b6fede451ebd3d91"
format_stamp: "Formatted at 2026-08-08 02:24:01 on dist-test-slave-pgkr"
server_key: "9fa34ee9564d0f4d0ca8c8dcff977c7d"
server_key_iv: "4cfdd1ee53cef46c7493c9f4d1158ef1"
server_key_version: "encryptionkey@0"
I20260808 02:24:01.937408 20400 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260808 02:24:01.942811 20415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:01.944413 20400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260808 02:24:01.944710 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "837f3578f0414e62b6fede451ebd3d91"
format_stamp: "Formatted at 2026-08-08 02:24:01 on dist-test-slave-pgkr"
server_key: "9fa34ee9564d0f4d0ca8c8dcff977c7d"
server_key_iv: "4cfdd1ee53cef46c7493c9f4d1158ef1"
server_key_version: "encryptionkey@0"
I20260808 02:24:01.945025 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:01.974997 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:01.982963 20400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:01.983377 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:02.025942 20400 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:40017
I20260808 02:24:02.026057 20468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:40017 every 8 connection(s)
I20260808 02:24:02.027887 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:02.032598 20469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:02.035266 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 20400
I20260808 02:24:02.035804 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:02.039543 20380 external_mini_cluster.cc:1428] Setting key b58964c37c6725672682e2f6d5bd5657
I20260808 02:24:02.051542 20469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91: Bootstrap starting.
I20260808 02:24:02.056237 20469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:02.057946 20469 log.cc:824] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:02.063359 20469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91: No bootstrap required, opened a new log
I20260808 02:24:02.077584 20469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837f3578f0414e62b6fede451ebd3d91" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40017 } }
I20260808 02:24:02.078197 20469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:02.078392 20469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837f3578f0414e62b6fede451ebd3d91, State: Initialized, Role: FOLLOWER
I20260808 02:24:02.079349 20469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837f3578f0414e62b6fede451ebd3d91" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40017 } }
I20260808 02:24:02.079831 20469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:02.080093 20469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:02.080374 20469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 02:24:02 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:02.083926 20469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837f3578f0414e62b6fede451ebd3d91" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40017 } }
I20260808 02:24:02.084710 20469 leader_election.cc:302] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 837f3578f0414e62b6fede451ebd3d91; no voters:
I20260808 02:24:02.086100 20469 leader_election.cc:288] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:02.086442 20477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:02.089304 20477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [term 1 LEADER]: Becoming Leader. State: Replica: 837f3578f0414e62b6fede451ebd3d91, State: Running, Role: LEADER
I20260808 02:24:02.090468 20477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837f3578f0414e62b6fede451ebd3d91" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40017 } }
I20260808 02:24:02.090966 20469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:02.095098 20478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "837f3578f0414e62b6fede451ebd3d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837f3578f0414e62b6fede451ebd3d91" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40017 } } }
I20260808 02:24:02.094978 20479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 837f3578f0414e62b6fede451ebd3d91. Latest consensus state: current_term: 1 leader_uuid: "837f3578f0414e62b6fede451ebd3d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837f3578f0414e62b6fede451ebd3d91" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40017 } } }
I20260808 02:24:02.096482 20478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:02.096493 20479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:02.099704 20482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:02.106398 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:02.045770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.19.231.62:40017)
0808 02:24:02.047749 (+ 1979us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:02.047768 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:02.049558 (+ 1790us) server_negotiation.cc:407] Connection header received
0808 02:24:02.064946 (+ 15388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:02.064979 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:02.065630 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:02.066263 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:02.070077 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:02.073140 (+ 3063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:02.075595 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:02.077095 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:02.077302 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:02.087446 (+ 10144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:02.087476 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:02.087524 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:02.087604 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:02.096420 (+ 8816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:02.098062 (+ 1642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:02.098084 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:02.098099 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:02.098528 (+ 429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:02.099170 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:02.099179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:02.099185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:02.101458 (+ 2273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:02.102274 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:02.103483 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0808 02:24:02.104142 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507,"thread_start_us":1059,"threads_started":1}
I20260808 02:24:02.118119 20482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:02.130214 20482 catalog_manager.cc:1347] Generated new cluster ID: 34a69a98072c4647b0a5499f2fd5b0c9
I20260808 02:24:02.130409 20482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:02.149539 20482 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:02.152143 20482 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:02.166818 20482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 837f3578f0414e62b6fede451ebd3d91: Generated new TSK 0
I20260808 02:24:02.167824 20482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:02.228317 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40017
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:02.845678 20498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:02.846115 20498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:02.846396 20498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:02.855305 20498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:02.855594 20498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:02.855850 20498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:02.868742 20498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:02.870699 20498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:02.872460 20498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:02.894807 20504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:02.896042 20508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:02.896129 20505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:02.897596 20498 server_base.cc:1034] running on GCE node
I20260808 02:24:02.898409 20498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:02.899878 20498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:02.901218 20498 hybrid_clock.cc:648] HybridClock initialized: now 1786155842901104 us; error 49 us; skew 500 ppm
I20260808 02:24:02.901609 20498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:02 dist-test-slave-pgkr krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155842, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:02.906571 20498 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:02.908663 20498 webserver.cc:460] Webserver started at http://127.19.231.1:37871/ using document root <none> and password file <none>
I20260808 02:24:02.909488 20498 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:02.909680 20498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:02.910118 20498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:02.914110 20498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "2ef7f0e2d89748e4ac472fd64d567418"
format_stamp: "Formatted at 2026-08-08 02:24:02 on dist-test-slave-pgkr"
server_key: "dd27015e67a19279302e44b8b277a0c3"
server_key_iv: "19153b68b3a8a3b741774df2c87e7dfc"
server_key_version: "encryptionkey@0"
I20260808 02:24:02.915146 20498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "2ef7f0e2d89748e4ac472fd64d567418"
format_stamp: "Formatted at 2026-08-08 02:24:02 on dist-test-slave-pgkr"
server_key: "dd27015e67a19279302e44b8b277a0c3"
server_key_iv: "19153b68b3a8a3b741774df2c87e7dfc"
server_key_version: "encryptionkey@0"
I20260808 02:24:02.922538 20498 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260808 02:24:02.927842 20516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:02.929369 20498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:24:02.929557 20498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "2ef7f0e2d89748e4ac472fd64d567418"
format_stamp: "Formatted at 2026-08-08 02:24:02 on dist-test-slave-pgkr"
server_key: "dd27015e67a19279302e44b8b277a0c3"
server_key_iv: "19153b68b3a8a3b741774df2c87e7dfc"
server_key_version: "encryptionkey@0"
I20260808 02:24:02.929832 20498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:02.947917 20498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:02.954358 20498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:02.954892 20498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:02.956620 20498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:02.958768 20498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:02.958931 20498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:02.959230 20498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:02.959396 20498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:03.066502 20498 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:39953
I20260808 02:24:03.066813 20633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:39953 every 8 connection(s)
I20260808 02:24:03.068368 20498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:03.069132 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 20498
I20260808 02:24:03.069608 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:03.070973 20380 external_mini_cluster.cc:1428] Setting key f70d2b744d8bb8531a046e92985d8ae9
Aug 08 02:24:03 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155842, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
I20260808 02:24:03.103070 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:03.073563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:53639 (local address 127.19.231.62:40017)
0808 02:24:03.074594 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:03.074599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:03.076442 (+ 1843us) server_negotiation.cc:407] Connection header received
0808 02:24:03.079922 (+ 3480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:03.079928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:03.080023 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:03.080199 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:03.082945 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.084833 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.087019 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.088023 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.088186 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:03.094381 (+ 6195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:03.094387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:03.094391 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:03.094426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:03.098413 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:03.099345 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:03.099352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:03.099358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:03.099561 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:03.100192 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:03.100200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:03.100206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:03.100834 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:03.101010 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:03.101863 (+ 853us) server_negotiation.cc:299] Negotiation successful
0808 02:24:03.102169 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":753,"threads_started":1}
I20260808 02:24:03.103935 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:03.074523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40017 (local address 127.19.231.1:53639)
0808 02:24:03.076270 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:03.076312 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:03.079635 (+ 3323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:03.080602 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:03.080611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:03.081349 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:03.082573 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:03.082672 (+ 99us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.085012 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.085017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:03.086872 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:03.086885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.088333 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.088339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:03.088734 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:03.089776 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:03.089801 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:03.094000 (+ 4199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:03.098593 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:03.098624 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:03.098647 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:03.099201 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:03.099731 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:03.099734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:03.099737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:03.100047 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:03.100974 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:03.100980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:03.101380 (+ 400us) client_negotiation.cc:769] Sending connection context
0808 02:24:03.101705 (+ 325us) client_negotiation.cc:240] Negotiation successful
0808 02:24:03.102166 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":925,"threads_started":1}
I20260808 02:24:03.105906 20634 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40017
I20260808 02:24:03.106372 20634 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:03.107738 20634 heartbeater.cc:511] Master 127.19.231.62:40017 requested a full tablet report, sending...
I20260808 02:24:03.110509 20432 ts_manager.cc:194] Registered new tserver with Master: 2ef7f0e2d89748e4ac472fd64d567418 (127.19.231.1:39953)
I20260808 02:24:03.112983 20432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:53639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:03.126211 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40017
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:03.780092 20640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:03.780550 20640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:03.780834 20640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:03.790428 20640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:03.790705 20640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:03.791153 20640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:03.804117 20640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:03.805717 20640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:03.807806 20640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:03.828461 20645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:03.829363 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:03.829490 20647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:03.831396 20640 server_base.cc:1034] running on GCE node
I20260808 02:24:03.832443 20640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:03.833645 20640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:03.834966 20640 hybrid_clock.cc:648] HybridClock initialized: now 1786155843834834 us; error 75 us; skew 500 ppm
I20260808 02:24:03.835510 20640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:03 dist-test-slave-pgkr krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155843, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:03.840638 20640 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:03.842932 20640 webserver.cc:460] Webserver started at http://127.19.231.2:37835/ using document root <none> and password file <none>
I20260808 02:24:03.843905 20640 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:03.844135 20640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:03.844524 20640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:03.848480 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "2b63a8713666424a9dd8a3b2dde8de9e"
format_stamp: "Formatted at 2026-08-08 02:24:03 on dist-test-slave-pgkr"
server_key: "ec76906d5fcc2bf62b13c9edd554a4d4"
server_key_iv: "fa13459bdd34b522f7c8f1c87ec16f22"
server_key_version: "encryptionkey@0"
I20260808 02:24:03.849368 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "2b63a8713666424a9dd8a3b2dde8de9e"
format_stamp: "Formatted at 2026-08-08 02:24:03 on dist-test-slave-pgkr"
server_key: "ec76906d5fcc2bf62b13c9edd554a4d4"
server_key_iv: "fa13459bdd34b522f7c8f1c87ec16f22"
server_key_version: "encryptionkey@0"
I20260808 02:24:03.856082 20640 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 02:24:03.860852 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:03.862418 20640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 02:24:03.862779 20640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "2b63a8713666424a9dd8a3b2dde8de9e"
format_stamp: "Formatted at 2026-08-08 02:24:03 on dist-test-slave-pgkr"
server_key: "ec76906d5fcc2bf62b13c9edd554a4d4"
server_key_iv: "fa13459bdd34b522f7c8f1c87ec16f22"
server_key_version: "encryptionkey@0"
I20260808 02:24:03.863188 20640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:03.885669 20640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:03.891705 20640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:03.892005 20640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:03.893463 20640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:03.895501 20640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:03.895645 20640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:03.895865 20640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:03.895969 20640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:03.967829 20640 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:37765
I20260808 02:24:03.968050 20784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:37765 every 8 connection(s)
I20260808 02:24:03.969429 20640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:03.971185 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 20640
I20260808 02:24:03.971459 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:03.971952 20380 external_mini_cluster.cc:1428] Setting key c65cba4775e601dc0139e3c7ff7e8efe
Aug 08 02:24:03 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155843, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
I20260808 02:24:03.996896 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:03.973554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:58701 (local address 127.19.231.62:40017)
0808 02:24:03.974273 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:03.974280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:03.975633 (+ 1353us) server_negotiation.cc:407] Connection header received
0808 02:24:03.978298 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:03.978304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:03.978393 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:03.978511 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:03.980361 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.981279 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.983362 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.983952 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.984122 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:03.989134 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:03.989141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:03.989144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:03.989182 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:03.993039 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:03.993870 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:03.993882 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:03.993885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:03.994067 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:03.994696 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:03.994706 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:03.994708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:03.995213 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:03.995366 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:03.996213 (+ 847us) server_negotiation.cc:299] Negotiation successful
0808 02:24:03.996478 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":448,"threads_started":1}
I20260808 02:24:03.998705 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:03.974222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40017 (local address 127.19.231.2:58701)
0808 02:24:03.975484 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:03.975523 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:03.978018 (+ 2495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:03.978813 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:03.978822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:03.979446 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:03.980190 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:03.980209 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.981460 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.981464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:03.983214 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:03.983226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:03.984330 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:03.984343 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:03.984613 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:03.985355 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:03.985402 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:03.988823 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:03.993190 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:03.993199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:03.993216 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:03.993740 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:03.994195 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:03.994200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:03.994204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:03.994481 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:03.995348 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:03.995355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:03.995789 (+ 434us) client_negotiation.cc:769] Sending connection context
0808 02:24:03.996059 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 02:24:03.996660 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":663,"threads_started":1}
I20260808 02:24:04.000710 20785 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40017
I20260808 02:24:04.001184 20785 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:04.002519 20785 heartbeater.cc:511] Master 127.19.231.62:40017 requested a full tablet report, sending...
I20260808 02:24:04.004630 20432 ts_manager.cc:194] Registered new tserver with Master: 2b63a8713666424a9dd8a3b2dde8de9e (127.19.231.2:37765)
I20260808 02:24:04.005761 20432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:58701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:04.022941 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40017
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:24:04.117686 20634 heartbeater.cc:503] Master 127.19.231.62:40017 was elected leader, sending a full tablet report...
W20260808 02:24:04.645411 20791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:04.645817 20791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:04.645977 20791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:04.656740 20791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:04.656953 20791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:04.657176 20791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:04.671046 20791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:04.672421 20791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:04.674054 20791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:04.693174 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:04.693356 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:04.693334 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:04.694749 20791 server_base.cc:1034] running on GCE node
I20260808 02:24:04.695431 20791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:04.696460 20791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:04.697713 20791 hybrid_clock.cc:648] HybridClock initialized: now 1786155844697648 us; error 51 us; skew 500 ppm
I20260808 02:24:04.698064 20791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:04 dist-test-slave-pgkr krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:04.702306 20791 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:04.704113 20791 webserver.cc:460] Webserver started at http://127.19.231.3:45367/ using document root <none> and password file <none>
I20260808 02:24:04.705041 20791 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:04.705286 20791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:04.705753 20791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:04.709450 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "0a181fca3d9f4df0a8be6f557cf5a5d6"
format_stamp: "Formatted at 2026-08-08 02:24:04 on dist-test-slave-pgkr"
server_key: "29e119a47148fdbd9c59fc532c696673"
server_key_iv: "3561b40dd5da3bd55fe775470ddb595d"
server_key_version: "encryptionkey@0"
I20260808 02:24:04.710383 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "0a181fca3d9f4df0a8be6f557cf5a5d6"
format_stamp: "Formatted at 2026-08-08 02:24:04 on dist-test-slave-pgkr"
server_key: "29e119a47148fdbd9c59fc532c696673"
server_key_iv: "3561b40dd5da3bd55fe775470ddb595d"
server_key_version: "encryptionkey@0"
I20260808 02:24:04.716907 20791 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260808 02:24:04.722076 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:04.723862 20791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 02:24:04.724121 20791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "0a181fca3d9f4df0a8be6f557cf5a5d6"
format_stamp: "Formatted at 2026-08-08 02:24:04 on dist-test-slave-pgkr"
server_key: "29e119a47148fdbd9c59fc532c696673"
server_key_iv: "3561b40dd5da3bd55fe775470ddb595d"
server_key_version: "encryptionkey@0"
I20260808 02:24:04.724457 20791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:04.741268 20791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:04.747627 20791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:04.748034 20791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:04.749609 20791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:04.751819 20791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:04.751983 20791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:04.752177 20791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:04.752298 20791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:04.830070 20791 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:35937
I20260808 02:24:04.830216 20928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:35937 every 8 connection(s)
I20260808 02:24:04.831770 20791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:04.840206 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 20791
I20260808 02:24:04.840549 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:04.841348 20380 external_mini_cluster.cc:1428] Setting key 03cb338e5b62d797b673d67906434c59
Aug 08 02:24:04 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:04.861202 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:04.835500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:38979 (local address 127.19.231.62:40017)
0808 02:24:04.836215 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:04.836220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:04.837762 (+ 1542us) server_negotiation.cc:407] Connection header received
0808 02:24:04.841348 (+ 3586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:04.841353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:04.841504 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:04.841602 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:04.844479 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.845771 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.847660 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.848315 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.848414 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:04.853021 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:04.853025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:04.853029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:04.853059 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:04.857059 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:04.858195 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:04.858203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:04.858210 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:04.858404 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:04.859079 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:04.859083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:04.859086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:04.859499 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:04.859660 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:04.860506 (+ 846us) server_negotiation.cc:299] Negotiation successful
0808 02:24:04.860734 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":428,"threads_started":1}
I20260808 02:24:04.862090 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:04.836271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40017 (local address 127.19.231.3:38979)
0808 02:24:04.837588 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:04.837636 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:04.841029 (+ 3393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:04.841958 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:04.841971 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:04.842990 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:04.844264 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:04.844294 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.845946 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.845952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:04.847533 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:04.847545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.848557 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.848563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:04.848886 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:04.849646 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:04.849684 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:04.852804 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:04.857246 (+ 4442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:04.857255 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:04.857273 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:04.858059 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:04.858538 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:04.858542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:04.858544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:04.858958 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:04.859615 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:04.859621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:04.860030 (+ 409us) client_negotiation.cc:769] Sending connection context
0808 02:24:04.860311 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 02:24:04.860751 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":594,"threads_started":1}
I20260808 02:24:04.863685 20929 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40017
I20260808 02:24:04.863996 20929 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:04.865103 20929 heartbeater.cc:511] Master 127.19.231.62:40017 requested a full tablet report, sending...
I20260808 02:24:04.866725 20432 ts_manager.cc:194] Registered new tserver with Master: 0a181fca3d9f4df0a8be6f557cf5a5d6 (127.19.231.3:35937)
I20260808 02:24:04.867674 20432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:38979
I20260808 02:24:04.871491 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:24:04 dist-test-slave-pgkr krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:24:04 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:04.934430 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:04.912936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.19.231.1:39953)
0808 02:24:04.913876 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:04.913902 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:04.914015 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:24:04.914333 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:04.914342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:04.914601 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:04.914845 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:04.917095 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.918559 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.920004 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.920711 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.920824 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:04.926158 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:04.926171 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:04.926186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:04.926288 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:04.931030 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:04.931801 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:04.931809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:04.931812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:04.931999 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:04.932516 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:04.932519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:04.932522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:04.932921 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:04.933260 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:04.933438 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 02:24:04.933725 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":466,"threads_started":1}
W20260808 02:24:04.936143 20547 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53412
Aug 08 02:24:04 dist-test-slave-pgkr krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:24:04 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:04.993067 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:04.974892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.19.231.1:39953)
0808 02:24:04.975172 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:04.975180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:04.975339 (+ 159us) server_negotiation.cc:407] Connection header received
0808 02:24:04.975558 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:04.975567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:04.975635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:04.975744 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:04.977754 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.978483 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.980030 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:04.980468 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:04.980619 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:04.985892 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:04.985898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:04.985901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:04.985942 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:04.989991 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:04.990791 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:04.990795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:04.990798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:04.990983 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:04.991443 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:04.991447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:04.991450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:04.991786 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:04.992008 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:04.992244 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 02:24:04.992583 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:24:04.995067 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 20498
I20260808 02:24:05.009087 20785 heartbeater.cc:503] Master 127.19.231.62:40017 was elected leader, sending a full tablet report...
I20260808 02:24:05.014321 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:04.996567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.19.231.1:39953)
0808 02:24:04.996787 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:04.996792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:04.996811 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:24:04.996944 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:04.996948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:04.997019 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:04.997104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:04.998900 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:05.000078 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:05.001993 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:05.002952 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:05.003095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:05.006242 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:05.006247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:05.006251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:05.006294 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:05.011258 (+ 4964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:05.011951 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:05.011955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:05.011958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:05.012219 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:05.012629 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:05.012633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:05.012636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:05.013043 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:05.013291 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:05.013492 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 02:24:05.013679 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:24:05.015609 20629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:05.102587 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 20498
I20260808 02:24:05.125845 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 20640
Aug 08 02:24:05 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:05.149787 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:05.127071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.19.231.2:37765)
0808 02:24:05.127702 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:05.127709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:05.127736 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:24:05.127797 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:05.127805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:05.128016 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:05.128183 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:05.129968 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:05.131207 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:05.132556 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:05.133110 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:05.133222 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:05.143610 (+ 10388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:05.143623 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:05.143643 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:05.143680 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:05.147014 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:05.147745 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:05.147752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:05.147755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:05.147873 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:05.148320 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:05.148323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:05.148325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:05.148672 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:05.148905 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:05.149207 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:24:05.149485 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":338,"threads_started":1}
I20260808 02:24:05.151415 20780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:05.236050 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 20640
I20260808 02:24:05.259990 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 20791
Aug 08 02:24:05 dist-test-slave-pgkr krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:05.278115 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:05.261132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.19.231.3:35937)
0808 02:24:05.261709 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:05.261716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:05.261749 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:24:05.261816 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:05.261823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:05.262057 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:05.262256 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:05.263783 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:05.264801 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:05.266226 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:05.266884 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:05.267069 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:05.271747 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:05.271760 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:05.271775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:05.271811 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:05.274903 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:05.275721 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:05.275730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:05.275732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:05.275879 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:05.276298 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:05.276305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:05.276309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:05.276701 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:05.277054 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:05.277414 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 02:24:05.277704 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":265,"threads_started":1}
I20260808 02:24:05.279628 20924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:05.364486 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 20791
I20260808 02:24:05.386029 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 20400
I20260808 02:24:05.387389 20464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:05.492175 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 20400
2026-08-08T02:24:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5542 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:05 dist-test-slave-pgkr krb5kdc[20952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:05 dist-test-slave-pgkr krb5kdc[20952](info): set up 2 sockets
Aug 08 02:24:05 dist-test-slave-pgkr krb5kdc[20952](info): commencing operation
krb5kdc: starting...
W20260808 02:24:06.636003 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:06 dist-test-slave-pgkr krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:06.834869 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:33197
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:44639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:07.170943 20965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:07.171221 20965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:07.171311 20965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:07.178427 20965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:07.178562 20965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:07.178609 20965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:07.178730 20965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:07.178800 20965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:07.188598 20965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:44639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:33197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:07.189924 20965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:07.191280 20965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:07.203527 20974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:07.203576 20971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:07.203599 20970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:07.204604 20965 server_base.cc:1034] running on GCE node
I20260808 02:24:07.205441 20965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:07.206939 20965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:07.208166 20965 hybrid_clock.cc:648] HybridClock initialized: now 1786155847208114 us; error 51 us; skew 500 ppm
I20260808 02:24:07.208523 20965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:07 dist-test-slave-pgkr krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:07.211757 20965 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:07.213342 20965 webserver.cc:460] Webserver started at http://127.19.231.62:41755/ using document root <none> and password file <none>
I20260808 02:24:07.214087 20965 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:07.214246 20965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:07.214577 20965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:07.217540 20965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "e2162979d9934c74bd600ee60fe15e34"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-pgkr"
server_key: "49f0e057ee2b6e2e8c973f00493154d7"
server_key_iv: "2312b85e519157f19138092df13ffea4"
server_key_version: "encryptionkey@0"
I20260808 02:24:07.218348 20965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "e2162979d9934c74bd600ee60fe15e34"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-pgkr"
server_key: "49f0e057ee2b6e2e8c973f00493154d7"
server_key_iv: "2312b85e519157f19138092df13ffea4"
server_key_version: "encryptionkey@0"
I20260808 02:24:07.223763 20965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 02:24:07.227257 20981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:07.228561 20965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:07.228766 20965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "e2162979d9934c74bd600ee60fe15e34"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-pgkr"
server_key: "49f0e057ee2b6e2e8c973f00493154d7"
server_key_iv: "2312b85e519157f19138092df13ffea4"
server_key_version: "encryptionkey@0"
I20260808 02:24:07.228962 20965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:07.240341 20965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:07.244765 20965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:07.245143 20965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:07.273649 20965 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:33197
I20260808 02:24:07.273671 21036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:33197 every 8 connection(s)
I20260808 02:24:07.275187 20965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:07.279028 21037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:07.285535 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 20965
I20260808 02:24:07.285776 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:07.286412 20380 external_mini_cluster.cc:1428] Setting key 63daca7dc4014404a6bd152a631b7efd
I20260808 02:24:07.289605 21037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34: Bootstrap starting.
I20260808 02:24:07.293073 21037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:07.294314 21037 log.cc:824] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:07.297565 21037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34: No bootstrap required, opened a new log
Aug 08 02:24:07 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:07.306285 21037 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2162979d9934c74bd600ee60fe15e34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33197 } }
I20260808 02:24:07.306701 21037 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:07.306841 21037 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2162979d9934c74bd600ee60fe15e34, State: Initialized, Role: FOLLOWER
I20260808 02:24:07.307507 21037 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2162979d9934c74bd600ee60fe15e34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33197 } }
I20260808 02:24:07.307803 21037 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:07.307946 21037 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:07.308131 21037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:07.310760 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:07.288465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.19.231.62:33197)
0808 02:24:07.289332 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:07.289345 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:07.289386 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:24:07.291257 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:07.291281 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:07.291672 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:07.291969 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:07.293899 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.295752 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.297464 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.298173 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.298309 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:07.302932 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:07.302940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:07.302957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:07.302996 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:07.306097 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:07.306802 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:07.306810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:07.306819 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:07.307040 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:07.307487 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:07.307497 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:07.307504 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:07.308243 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:07.308596 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:07.308932 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 02:24:07.309381 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":390,"threads_started":1}
I20260808 02:24:07.311522 21037 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2162979d9934c74bd600ee60fe15e34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33197 } }
I20260808 02:24:07.312104 21037 leader_election.cc:302] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2162979d9934c74bd600ee60fe15e34; no voters:
I20260808 02:24:07.312801 21037 leader_election.cc:288] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:07.312918 21042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:07.314867 21042 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [term 1 LEADER]: Becoming Leader. State: Replica: e2162979d9934c74bd600ee60fe15e34, State: Running, Role: LEADER
I20260808 02:24:07.315235 21037 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:07.315402 21042 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2162979d9934c74bd600ee60fe15e34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33197 } }
I20260808 02:24:07.319801 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2162979d9934c74bd600ee60fe15e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2162979d9934c74bd600ee60fe15e34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33197 } } }
I20260808 02:24:07.320171 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:07.320353 21044 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2162979d9934c74bd600ee60fe15e34. Latest consensus state: current_term: 1 leader_uuid: "e2162979d9934c74bd600ee60fe15e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2162979d9934c74bd600ee60fe15e34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33197 } } }
I20260808 02:24:07.320798 21044 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:07.321242 21052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:07.326696 21052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:07.335366 21052 catalog_manager.cc:1347] Generated new cluster ID: b5d54526a65d4f74a63bc92a00c7a4de
I20260808 02:24:07.335481 21052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:07.347352 21052 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:07.349175 21052 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:07.356060 21052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2162979d9934c74bd600ee60fe15e34: Generated new TSK 0
I20260808 02:24:07.357005 21052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:07.412492 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33197
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:44639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:07.754945 21065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:07.755236 21065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:07.755427 21065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:07.762915 21065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:07.763058 21065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:07.763262 21065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:07.773079 21065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:44639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:07.774426 21065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:07.775869 21065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:07.789258 21070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:07.789207 21071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:07.789292 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:07.790354 21065 server_base.cc:1034] running on GCE node
I20260808 02:24:07.791265 21065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:07.792490 21065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:07.793700 21065 hybrid_clock.cc:648] HybridClock initialized: now 1786155847793664 us; error 34 us; skew 500 ppm
I20260808 02:24:07.794040 21065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:07 dist-test-slave-pgkr krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:07.797412 21065 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:07.799027 21065 webserver.cc:460] Webserver started at http://127.19.231.1:35753/ using document root <none> and password file <none>
I20260808 02:24:07.799755 21065 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:07.799860 21065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:07.800122 21065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:07.803066 21065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "377e1cca28fc43abaf9e0d090ac4e839"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-pgkr"
server_key: "0252ab7a5a2bd3ee32f8b715ac1d3d96"
server_key_iv: "4509ef7d14bf614337b666c4f05afab7"
server_key_version: "encryptionkey@0"
I20260808 02:24:07.803826 21065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "377e1cca28fc43abaf9e0d090ac4e839"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-pgkr"
server_key: "0252ab7a5a2bd3ee32f8b715ac1d3d96"
server_key_iv: "4509ef7d14bf614337b666c4f05afab7"
server_key_version: "encryptionkey@0"
I20260808 02:24:07.808806 21065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:07.812677 21081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:07.814054 21065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:07.814216 21065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "377e1cca28fc43abaf9e0d090ac4e839"
format_stamp: "Formatted at 2026-08-08 02:24:07 on dist-test-slave-pgkr"
server_key: "0252ab7a5a2bd3ee32f8b715ac1d3d96"
server_key_iv: "4509ef7d14bf614337b666c4f05afab7"
server_key_version: "encryptionkey@0"
I20260808 02:24:07.814417 21065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:07.840703 21065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:07.845129 21065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:07.845449 21065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:07.846448 21065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:07.847877 21065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:07.847952 21065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:07.848120 21065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:07.848183 21065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:07.882539 21065 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:36409
I20260808 02:24:07.882678 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:36409 every 8 connection(s)
I20260808 02:24:07.883930 21065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:07.893342 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 21065
I20260808 02:24:07.893597 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:07.894217 20380 external_mini_cluster.cc:1428] Setting key 287881507001f9c418d29d3f863717bc
Aug 08 02:24:07 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:07.906044 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:07.886578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:58305 (local address 127.19.231.62:33197)
0808 02:24:07.887136 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:07.887143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:07.888358 (+ 1215us) server_negotiation.cc:407] Connection header received
0808 02:24:07.890192 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:07.890197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:07.890273 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:07.890392 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:07.891930 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.892743 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.894549 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.895128 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.895273 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:07.898976 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:07.898985 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:07.898992 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:07.899050 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:07.902567 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:07.903676 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:07.903680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:07.903682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:07.903804 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:07.904303 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:07.904306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:07.904308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:07.904574 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:07.904686 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:07.905606 (+ 920us) server_negotiation.cc:299] Negotiation successful
0808 02:24:07.905773 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":231,"threads_started":1}
I20260808 02:24:07.907625 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:07.887334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:33197 (local address 127.19.231.1:58305)
0808 02:24:07.888178 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:07.888226 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:07.889909 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:07.890588 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:07.890596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:07.891197 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:07.891752 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:07.891768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.892885 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.892889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:07.894401 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:07.894418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:07.895357 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:07.895364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:07.895656 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:07.896443 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:07.896465 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:07.898753 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:07.902777 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:07.902785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:07.902818 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:07.903498 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:07.903916 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:07.903923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:07.903927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:07.904188 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:07.904711 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:07.904722 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:07.905174 (+ 452us) client_negotiation.cc:769] Sending connection context
0808 02:24:07.905474 (+ 300us) client_negotiation.cc:240] Negotiation successful
0808 02:24:07.906068 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":273,"threads_started":1}
I20260808 02:24:07.909384 21199 heartbeater.cc:347] Connected to a master server at 127.19.231.62:33197
I20260808 02:24:07.909746 21199 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:07.910653 21199 heartbeater.cc:511] Master 127.19.231.62:33197 requested a full tablet report, sending...
I20260808 02:24:07.912897 20999 ts_manager.cc:194] Registered new tserver with Master: 377e1cca28fc43abaf9e0d090ac4e839 (127.19.231.1:36409)
I20260808 02:24:07.914402 20999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:58305
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:07.942481 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33197
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:44639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:08.321252 21206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:08.321709 21206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:08.322000 21206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:08.332119 21206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:08.332314 21206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:08.332567 21206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:08.345578 21206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:44639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:08.347074 21206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:08.348737 21206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:08.368563 21211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.368563 21212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.368721 21214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:08.370426 21206 server_base.cc:1034] running on GCE node
I20260808 02:24:08.371300 21206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:08.372591 21206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:08.373916 21206 hybrid_clock.cc:648] HybridClock initialized: now 1786155848373827 us; error 69 us; skew 500 ppm
I20260808 02:24:08.374400 21206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:08 dist-test-slave-pgkr krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:08.379333 21206 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:08.381377 21206 webserver.cc:460] Webserver started at http://127.19.231.2:40067/ using document root <none> and password file <none>
I20260808 02:24:08.382217 21206 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:08.382452 21206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:08.382933 21206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:08.386731 21206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "a324d501a8724e16ae071535b812c929"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-pgkr"
server_key: "d052e974cf76ce4975e60b6041d45348"
server_key_iv: "2bd859c5c1372cfab8efc05b2b74700c"
server_key_version: "encryptionkey@0"
I20260808 02:24:08.387790 21206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "a324d501a8724e16ae071535b812c929"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-pgkr"
server_key: "d052e974cf76ce4975e60b6041d45348"
server_key_iv: "2bd859c5c1372cfab8efc05b2b74700c"
server_key_version: "encryptionkey@0"
I20260808 02:24:08.394690 21206 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260808 02:24:08.399770 21222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.401701 21206 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:24:08.402002 21206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "a324d501a8724e16ae071535b812c929"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-pgkr"
server_key: "d052e974cf76ce4975e60b6041d45348"
server_key_iv: "2bd859c5c1372cfab8efc05b2b74700c"
server_key_version: "encryptionkey@0"
I20260808 02:24:08.402379 21206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:08.419052 21206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:08.425565 21206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:08.426108 21206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:08.428007 21206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:08.430589 21206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:08.430788 21206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 02:24:08.431111 21206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:08.431281 21206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.518142 21206 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:34807
I20260808 02:24:08.518193 21339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:34807 every 8 connection(s)
I20260808 02:24:08.519989 21206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:08.525548 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 21206
I20260808 02:24:08.525887 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:08.526731 20380 external_mini_cluster.cc:1428] Setting key fa78c35ee55ce4635fcc214a6bfe7962
Aug 08 02:24:08 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
I20260808 02:24:08.550335 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.524338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:42267 (local address 127.19.231.62:33197)
0808 02:24:08.525209 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:08.525214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:08.526866 (+ 1652us) server_negotiation.cc:407] Connection header received
0808 02:24:08.529686 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:08.529692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:08.529774 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:08.529891 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:08.532039 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.532941 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.535191 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.535704 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.535838 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.541614 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:08.541619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:08.541623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:08.541655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:08.545577 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.546811 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.546816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.546818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.547034 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:08.547860 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:08.547863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:08.547866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:08.548291 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:08.548408 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:08.549590 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0808 02:24:08.549779 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":650,"threads_started":1}
I20260808 02:24:08.551275 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:08.525203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:33197 (local address 127.19.231.2:42267)
0808 02:24:08.526596 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:08.526698 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:08.529420 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:08.530182 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:08.530191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:08.531023 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:08.531822 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.531842 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.533215 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.533219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.535056 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:08.535068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:08.536073 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:08.536086 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:08.536559 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:08.537567 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:08.537636 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:08.541284 (+ 3648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:08.545750 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.545759 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.545788 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.546661 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.547187 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:08.547194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:08.547201 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:08.547740 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:08.548471 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:08.548480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:08.549073 (+ 593us) client_negotiation.cc:769] Sending connection context
0808 02:24:08.549406 (+ 333us) client_negotiation.cc:240] Negotiation successful
0808 02:24:08.549887 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":710,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
I20260808 02:24:08.553229 21340 heartbeater.cc:347] Connected to a master server at 127.19.231.62:33197
I20260808 02:24:08.553555 21340 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:08.554373 21340 heartbeater.cc:511] Master 127.19.231.62:33197 requested a full tablet report, sending...
I20260808 02:24:08.556169 20999 ts_manager.cc:194] Registered new tserver with Master: a324d501a8724e16ae071535b812c929 (127.19.231.2:34807)
I20260808 02:24:08.557093 20999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:42267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:08.577194 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33197
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:44639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:24:08.917989 21199 heartbeater.cc:503] Master 127.19.231.62:33197 was elected leader, sending a full tablet report...
W20260808 02:24:08.932821 21347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:08.933115 21347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:08.933355 21347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:08.940869 21347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:08.940989 21347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:08.941210 21347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:08.951301 21347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:44639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:08.952705 21347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:08.954150 21347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:08.967936 21357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.967973 21355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:08.967996 21354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:08.969288 21347 server_base.cc:1034] running on GCE node
I20260808 02:24:08.969825 21347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:08.970811 21347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:08.972009 21347 hybrid_clock.cc:648] HybridClock initialized: now 1786155848971953 us; error 46 us; skew 500 ppm
I20260808 02:24:08.972344 21347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:08 dist-test-slave-pgkr krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:08.975564 21347 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:08.977181 21347 webserver.cc:460] Webserver started at http://127.19.231.3:44919/ using document root <none> and password file <none>
I20260808 02:24:08.977870 21347 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:08.977991 21347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:08.978324 21347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:08.981338 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "9e8c5c3cc7084677b1baddc54bcaeb53"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-pgkr"
server_key: "d5c907aceb08100747cb5e0eb2bf25ca"
server_key_iv: "f90e23b6f48c1ff1b792580147a23cbb"
server_key_version: "encryptionkey@0"
I20260808 02:24:08.982143 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "9e8c5c3cc7084677b1baddc54bcaeb53"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-pgkr"
server_key: "d5c907aceb08100747cb5e0eb2bf25ca"
server_key_iv: "f90e23b6f48c1ff1b792580147a23cbb"
server_key_version: "encryptionkey@0"
I20260808 02:24:08.987555 21347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:24:08.991104 21364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:08.992486 21347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:24:08.992659 21347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "9e8c5c3cc7084677b1baddc54bcaeb53"
format_stamp: "Formatted at 2026-08-08 02:24:08 on dist-test-slave-pgkr"
server_key: "d5c907aceb08100747cb5e0eb2bf25ca"
server_key_iv: "f90e23b6f48c1ff1b792580147a23cbb"
server_key_version: "encryptionkey@0"
I20260808 02:24:08.992856 21347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:09.025701 21347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:09.030086 21347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:09.030395 21347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:09.031471 21347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:09.032960 21347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:09.033067 21347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:09.033219 21347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:09.033298 21347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:09.067669 21347 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:35073
I20260808 02:24:09.067771 21481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:35073 every 8 connection(s)
I20260808 02:24:09.069178 21347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:09.078819 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 21347
I20260808 02:24:09.079018 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:09.079434 20380 external_mini_cluster.cc:1428] Setting key ffe32d86c1223a2d6de1742498950fe0
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:09.089843 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.072171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:49555 (local address 127.19.231.62:33197)
0808 02:24:09.072681 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.072687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.073704 (+ 1017us) server_negotiation.cc:407] Connection header received
0808 02:24:09.075657 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.075662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.075744 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.075845 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.077313 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.078098 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.079839 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.080384 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.080517 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.084608 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.084613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.084616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.084651 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.087229 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.087950 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.087954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.087956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.088075 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.088432 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.088435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.088437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.088692 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.088806 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.089412 (+ 606us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.089565 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":239,"threads_started":1}
I20260808 02:24:09.090951 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.072858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:33197 (local address 127.19.231.3:49555)
0808 02:24:09.073541 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.073589 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:09.075397 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:09.076048 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:09.076056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:09.076564 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:09.077162 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.077179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.078237 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.078240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.079691 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.079705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.080691 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.080701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.081031 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.081866 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:09.081896 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:09.084407 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:09.087359 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:09.087366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:09.087379 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:09.087841 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:09.088160 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:09.088163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:09.088165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:09.088338 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:09.088795 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:09.088801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:09.089107 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 02:24:09.089332 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 02:24:09.089751 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":279,"threads_started":1}
I20260808 02:24:09.092417 21482 heartbeater.cc:347] Connected to a master server at 127.19.231.62:33197
I20260808 02:24:09.092785 21482 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:09.093657 21482 heartbeater.cc:511] Master 127.19.231.62:33197 requested a full tablet report, sending...
I20260808 02:24:09.095304 20999 ts_manager.cc:194] Registered new tserver with Master: 9e8c5c3cc7084677b1baddc54bcaeb53 (127.19.231.3:35073)
I20260808 02:24:09.096127 20999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:49555
I20260808 02:24:09.107126 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:24:09.127372 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.113762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.19.231.62:33197)
0808 02:24:09.113932 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.113937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.114370 (+ 433us) server_negotiation.cc:407] Connection header received
0808 02:24:09.114572 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.114577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.114690 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.114790 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.116379 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.117170 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.118532 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.119026 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.119180 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.121555 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.121561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.121566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.121605 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.124470 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.125324 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.125330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.125333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.125468 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.125841 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.125848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.125852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.126184 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.126428 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.126819 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.127044 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:24:09.133759 20999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:09.140722 20999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:09.164249 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.151969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:36409 (local address 127.0.0.1:43968)
0808 02:24:09.152686 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.152747 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:09.152929 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:09.156849 (+ 3920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:09.156857 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:09.156886 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:09.157234 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.157247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.158987 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.158995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.161535 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.161550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.163215 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.163222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.163405 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.163460 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:24:09.163610 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 02:24:09.163817 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":279,"threads_started":1}
I20260808 02:24:09.164934 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.152318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.19.231.1:36409)
0808 02:24:09.156189 (+ 3871us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.156197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.156225 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:24:09.156326 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.156335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.156628 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.156843 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:09.157450 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.158863 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.161697 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.163003 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.163168 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.163411 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.164430 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.164593 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3641,"thread_start_us":3462,"threads_started":1}
I20260808 02:24:09.165071 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.153111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:34807 (local address 127.0.0.1:47470)
0808 02:24:09.156489 (+ 3378us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.156513 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:09.156731 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:09.157628 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:09.157633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:09.157657 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:09.158060 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.158073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.160290 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.160297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.162154 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.162165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.164260 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.164268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.164432 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.164467 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:24:09.164571 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:24:09.164709 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3261,"thread_start_us":2221,"threads_started":1}
I20260808 02:24:09.165673 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.153932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.19.231.3:35073)
0808 02:24:09.155250 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.155262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.156696 (+ 1434us) server_negotiation.cc:407] Connection header received
0808 02:24:09.156821 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.156828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.157048 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.157271 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:09.157897 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.159548 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.162465 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.164133 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.164259 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.164464 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.165168 (+ 704us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.165351 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":867,"threads_started":1}
I20260808 02:24:09.165917 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.153526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.19.231.2:34807)
0808 02:24:09.154076 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.154087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.156623 (+ 2536us) server_negotiation.cc:407] Connection header received
0808 02:24:09.156933 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.156941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.157155 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.157341 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:09.158232 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.160163 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.162296 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.163771 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.163902 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.164080 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.165398 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.165570 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":536,"threads_started":1}
I20260808 02:24:09.165693 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.152952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:35073 (local address 127.0.0.1:55608)
0808 02:24:09.156430 (+ 3478us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.156455 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:09.156572 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:09.157270 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:09.157280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:09.157307 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:09.157699 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.157709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.159731 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.159738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.161459 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.161471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.164604 (+ 3133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.164610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.164785 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.164833 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:24:09.164917 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:24:09.165036 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3340,"mutex_wait_us":7,"thread_start_us":3180,"threads_started":1}
I20260808 02:24:09.169016 21130 tablet_service.cc:1467] Processing CreateTablet for tablet a1fc638c288645669c4ebae0d4947bab (DEFAULT_TABLE table=test-table [id=8d60e869c6eb4eea8ebaea16d39cfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:09.169503 21414 tablet_service.cc:1467] Processing CreateTablet for tablet a1fc638c288645669c4ebae0d4947bab (DEFAULT_TABLE table=test-table [id=8d60e869c6eb4eea8ebaea16d39cfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:09.169504 21272 tablet_service.cc:1467] Processing CreateTablet for tablet a1fc638c288645669c4ebae0d4947bab (DEFAULT_TABLE table=test-table [id=8d60e869c6eb4eea8ebaea16d39cfce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:09.170451 21130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fc638c288645669c4ebae0d4947bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:09.170923 21414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fc638c288645669c4ebae0d4947bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:09.170923 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fc638c288645669c4ebae0d4947bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:09.182399 21501 tablet_bootstrap.cc:492] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839: Bootstrap starting.
I20260808 02:24:09.184872 21502 tablet_bootstrap.cc:492] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53: Bootstrap starting.
I20260808 02:24:09.186316 21504 tablet_bootstrap.cc:492] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929: Bootstrap starting.
I20260808 02:24:09.186697 21501 tablet_bootstrap.cc:654] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:09.187791 21501 log.cc:824] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:09.188701 21502 tablet_bootstrap.cc:654] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:09.189831 21504 tablet_bootstrap.cc:654] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:09.189931 21502 log.cc:824] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:09.190699 21501 tablet_bootstrap.cc:492] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839: No bootstrap required, opened a new log
I20260808 02:24:09.191005 21501 ts_tablet_manager.cc:1397] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 02:24:09.191207 21504 log.cc:824] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:09.193444 21502 tablet_bootstrap.cc:492] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53: No bootstrap required, opened a new log
I20260808 02:24:09.193713 21502 ts_tablet_manager.cc:1397] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 02:24:09.194207 21504 tablet_bootstrap.cc:492] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929: No bootstrap required, opened a new log
I20260808 02:24:09.194453 21504 ts_tablet_manager.cc:1397] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 02:24:09.201140 21502 raft_consensus.cc:348] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.201583 21502 raft_consensus.cc:374] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:09.201706 21502 raft_consensus.cc:729] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8c5c3cc7084677b1baddc54bcaeb53, State: Initialized, Role: FOLLOWER
I20260808 02:24:09.202029 21501 raft_consensus.cc:348] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.202456 21501 raft_consensus.cc:374] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:09.202584 21501 raft_consensus.cc:729] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377e1cca28fc43abaf9e0d090ac4e839, State: Initialized, Role: FOLLOWER
I20260808 02:24:09.202790 21502 consensus_queue.cc:260] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.203217 21501 consensus_queue.cc:260] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.204047 21482 heartbeater.cc:503] Master 127.19.231.62:33197 was elected leader, sending a full tablet report...
I20260808 02:24:09.204545 21502 ts_tablet_manager.cc:1428] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260808 02:24:09.205106 21504 raft_consensus.cc:348] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.205673 21504 raft_consensus.cc:374] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:09.205799 21504 raft_consensus.cc:729] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a324d501a8724e16ae071535b812c929, State: Initialized, Role: FOLLOWER
I20260808 02:24:09.206497 21504 consensus_queue.cc:260] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.207841 21340 heartbeater.cc:503] Master 127.19.231.62:33197 was elected leader, sending a full tablet report...
I20260808 02:24:09.208777 21504 ts_tablet_manager.cc:1428] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260808 02:24:09.210547 21501 ts_tablet_manager.cc:1428] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839: Time spent starting tablet: real 0.019s user 0.010s sys 0.007s
W20260808 02:24:09.273887 21342 tablet.cc:2369] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:09.279827 21508 raft_consensus.cc:482] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:09.280153 21508 raft_consensus.cc:504] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.281972 21508 leader_election.cc:288] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 377e1cca28fc43abaf9e0d090ac4e839 (127.19.231.1:36409), a324d501a8724e16ae071535b812c929 (127.19.231.2:34807)
I20260808 02:24:09.289027 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.282999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:49859 (local address 127.19.231.2:34807)
0808 02:24:09.283201 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.283206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.283755 (+ 549us) server_negotiation.cc:407] Connection header received
0808 02:24:09.283911 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.283916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.283999 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.284091 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:09.284667 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.285447 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.287295 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.288050 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.288245 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.288320 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.288635 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.288765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:24:09.289417 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.282804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:34807 (local address 127.19.231.3:49859)
0808 02:24:09.283632 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.283649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:09.283773 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:09.284122 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:09.284127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:09.284160 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:09.284508 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.284517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.285616 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.285620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.287134 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.287146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.288252 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.288258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.288454 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.288491 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:24:09.288590 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 02:24:09.288741 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":45,"thread_start_us":529,"threads_started":1}
I20260808 02:24:09.289808 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.282387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:39509 (local address 127.19.231.1:36409)
0808 02:24:09.282575 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.282581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.282643 (+ 62us) server_negotiation.cc:407] Connection header received
0808 02:24:09.282954 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.282961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.283060 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.283171 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:09.283881 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.284813 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.287317 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.288551 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.288721 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.288790 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.289404 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.289549 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:24:09.289824 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.282318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:36409 (local address 127.19.231.3:39509)
0808 02:24:09.282539 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.282562 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:09.282740 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:09.283408 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:09.283417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:09.283444 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:09.283666 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.283674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.284946 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.284951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.287120 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:09.287135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.288878 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.288883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:09.289060 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.289092 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:24:09.289349 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 02:24:09.289500 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":27}
I20260808 02:24:09.290581 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fc638c288645669c4ebae0d4947bab" candidate_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a324d501a8724e16ae071535b812c929" is_pre_election: true
I20260808 02:24:09.290987 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fc638c288645669c4ebae0d4947bab" candidate_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377e1cca28fc43abaf9e0d090ac4e839" is_pre_election: true
I20260808 02:24:09.291275 21292 raft_consensus.cc:2454] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8c5c3cc7084677b1baddc54bcaeb53 in term 0.
I20260808 02:24:09.291587 21152 raft_consensus.cc:2454] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8c5c3cc7084677b1baddc54bcaeb53 in term 0.
I20260808 02:24:09.292034 21367 leader_election.cc:302] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e8c5c3cc7084677b1baddc54bcaeb53, a324d501a8724e16ae071535b812c929; no voters:
I20260808 02:24:09.292448 21508 raft_consensus.cc:2781] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:09.292645 21508 raft_consensus.cc:482] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:09.292764 21508 raft_consensus.cc:3037] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:09.295076 21508 raft_consensus.cc:504] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.295635 21508 leader_election.cc:288] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [CANDIDATE]: Term 1 election: Requested vote from peers 377e1cca28fc43abaf9e0d090ac4e839 (127.19.231.1:36409), a324d501a8724e16ae071535b812c929 (127.19.231.2:34807)
I20260808 02:24:09.296065 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fc638c288645669c4ebae0d4947bab" candidate_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377e1cca28fc43abaf9e0d090ac4e839"
I20260808 02:24:09.296187 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fc638c288645669c4ebae0d4947bab" candidate_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a324d501a8724e16ae071535b812c929"
I20260808 02:24:09.296294 21152 raft_consensus.cc:3037] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:09.296418 21292 raft_consensus.cc:3037] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:09.298671 21152 raft_consensus.cc:2454] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8c5c3cc7084677b1baddc54bcaeb53 in term 1.
I20260808 02:24:09.298771 21292 raft_consensus.cc:2454] T a1fc638c288645669c4ebae0d4947bab P a324d501a8724e16ae071535b812c929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8c5c3cc7084677b1baddc54bcaeb53 in term 1.
I20260808 02:24:09.299085 21367 leader_election.cc:302] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 377e1cca28fc43abaf9e0d090ac4e839, 9e8c5c3cc7084677b1baddc54bcaeb53; no voters:
I20260808 02:24:09.299355 21508 raft_consensus.cc:2781] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:09.299798 21508 raft_consensus.cc:686] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8c5c3cc7084677b1baddc54bcaeb53, State: Running, Role: LEADER
I20260808 02:24:09.300308 21508 consensus_queue.cc:237] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } }
I20260808 02:24:09.305524 20998 catalog_manager.cc:5495] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e8c5c3cc7084677b1baddc54bcaeb53 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "377e1cca28fc43abaf9e0d090ac4e839" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a324d501a8724e16ae071535b812c929" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8c5c3cc7084677b1baddc54bcaeb53" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 35073 } health_report { overall_health: HEALTHY } } }
W20260808 02:24:09.321029 21483 tablet.cc:2369] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:09.360158 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.344280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.19.231.1:36409)
0808 02:24:09.344499 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.344507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.344889 (+ 382us) server_negotiation.cc:407] Connection header received
0808 02:24:09.345095 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.345099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.345174 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.345283 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.346893 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.347665 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.348922 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.349357 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.349494 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.353535 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.353557 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.353582 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.353635 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.356996 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.357652 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.357661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.357663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.357873 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.358290 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.358295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.358298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.358799 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.359188 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.359486 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.359711 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 02:24:09.386785 21200 tablet.cc:2369] T a1fc638c288645669c4ebae0d4947bab P 377e1cca28fc43abaf9e0d090ac4e839: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:09.415277 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.398733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.19.231.1:36409)
0808 02:24:09.398942 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.398951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.399394 (+ 443us) server_negotiation.cc:407] Connection header received
0808 02:24:09.399614 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.399619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.399687 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.399795 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.401246 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.402130 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.403992 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.404524 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.404679 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.409281 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.409285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.409288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.409322 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.412344 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.412967 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.412973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.412975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.413169 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.413567 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.413571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.413573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.413891 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.414252 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.414493 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.414794 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:09.474776 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.454004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.19.231.1:36409)
0808 02:24:09.454310 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.454321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.454885 (+ 564us) server_negotiation.cc:407] Connection header received
0808 02:24:09.455092 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.455097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.455165 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.455254 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.457605 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.458338 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.460172 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.460870 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.460984 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.467479 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.467485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.467488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.467525 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.471445 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.472242 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.472247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.472250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.472495 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.472952 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.472957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.472959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.473386 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.473707 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.473910 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.474147 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260808 02:24:09.477049 21110 tablet_service.cc:2908] Rejecting scan request for tablet a1fc638c288645669c4ebae0d4947bab: Uninitialized: clean time has not yet been initialized
I20260808 02:24:09.480940 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21065
I20260808 02:24:09.497591 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.482090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.19.231.1:36409)
0808 02:24:09.482298 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.482302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.482806 (+ 504us) server_negotiation.cc:407] Connection header received
0808 02:24:09.482937 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.482941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.483031 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.483123 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.484280 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.484979 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.486185 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.486976 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.487151 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.489832 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.489837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.489841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.489876 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.494086 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.494898 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.494902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.494905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.495055 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.495407 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.495410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.495412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.495863 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.496357 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.496701 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.496982 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:24:09.498508 21194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:09.593011 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 21065
I20260808 02:24:09.615540 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21206
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:09.634814 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.616392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.19.231.2:34807)
0808 02:24:09.616590 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.616597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.616681 (+ 84us) server_negotiation.cc:407] Connection header received
0808 02:24:09.616899 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.616910 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.616983 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.617195 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.619217 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.620706 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.622658 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.623203 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.623369 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.628860 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.628876 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.628893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.628928 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.631923 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.632677 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.632684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.632687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.632817 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.633192 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.633198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.633201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.633581 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.633935 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.634183 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.634416 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:24:09.635591 21335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:09.717550 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.717057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:36409 (local address 127.19.231.3:51983)
0808 02:24:09.717237 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.717380 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:24:09.717947 21367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.1:36409: connect: Connection refused (error 111)
I20260808 02:24:09.719558 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.719231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:36409 (local address 127.19.231.3:43787)
0808 02:24:09.719380 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:09.719452 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:24:09.719817 21367 consensus_peers.cc:487] T a1fc638c288645669c4ebae0d4947bab P 9e8c5c3cc7084677b1baddc54bcaeb53 -> Peer 377e1cca28fc43abaf9e0d090ac4e839 (127.19.231.1:36409): Couldn't send request to peer 377e1cca28fc43abaf9e0d090ac4e839. Status: Network error: Client connection negotiation failed: client connection to 127.19.231.1:36409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:24:09.725530 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 21206
I20260808 02:24:09.748512 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21347
Aug 08 02:24:09 dist-test-slave-pgkr krb5kdc[20952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:09.764168 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:09.749269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.19.231.3:35073)
0808 02:24:09.749782 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:09.749787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:09.749809 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:24:09.749866 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:09.749869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:09.749934 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:09.750028 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:09.751349 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.752089 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.753327 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:09.753748 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:09.753843 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:09.758174 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:09.758192 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:09.758209 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:09.758248 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:09.761473 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.762128 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.762136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.762139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.762303 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:09.762692 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:09.762696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:09.762698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:09.763059 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:09.763339 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:09.763547 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 02:24:09.763751 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":285,"threads_started":1}
I20260808 02:24:09.764973 21476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:09.854873 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 21347
I20260808 02:24:09.875356 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 20965
I20260808 02:24:09.876655 21032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:09.990229 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 20965
2026-08-08T02:24:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4501 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:10 dist-test-slave-pgkr krb5kdc[21535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:10 dist-test-slave-pgkr krb5kdc[21535](info): set up 2 sockets
Aug 08 02:24:10 dist-test-slave-pgkr krb5kdc[21535](info): commencing operation
krb5kdc: starting...
W20260808 02:24:11.225812 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260808 02:24:11.431255 20380 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.205s user 0.000s sys 0.006s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260808 02:24:11.697535 20380 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.266s user 0.000s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 02:24:12.015594 20380 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.318s user 0.000s sys 0.006s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/test-user.keytab.
W20260808 02:24:12.498907 20380 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.483s user 0.000s sys 0.006s
Aug 08 02:24:12 dist-test-slave-pgkr krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:12.595145 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:34655
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:34655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:12.948990 21548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:12.949360 21548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:12.949517 21548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:12.957023 21548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:12.957171 21548 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:24:12.957273 21548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:12.957391 21548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:12.957475 21548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:12.957576 21548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:12.968098 21548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:34655
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:34655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:12.969486 21548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:12.971045 21548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:12.983225 21555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:12.983225 21556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:12.983780 21559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:12.984555 21548 server_base.cc:1034] running on GCE node
I20260808 02:24:12.985438 21548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:12.987077 21548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:12.988358 21548 hybrid_clock.cc:648] HybridClock initialized: now 1786155852988294 us; error 40 us; skew 500 ppm
I20260808 02:24:12.988741 21548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:12 dist-test-slave-pgkr krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155852, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:12.992028 21548 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:12.993592 21548 webserver.cc:460] Webserver started at http://127.19.231.62:33603/ using document root <none> and password file <none>
I20260808 02:24:12.994282 21548 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:12.994370 21548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:12.994728 21548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:12.997707 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "b6f4714804e04741b7164f282f9d039c"
format_stamp: "Formatted at 2026-08-08 02:24:12 on dist-test-slave-pgkr"
server_key: "97619ced51c33688a79dd6db97ec9d30"
server_key_iv: "7b7a39655eb90f6d884a6c4601640054"
server_key_version: "encryptionkey@0"
I20260808 02:24:12.998421 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "b6f4714804e04741b7164f282f9d039c"
format_stamp: "Formatted at 2026-08-08 02:24:12 on dist-test-slave-pgkr"
server_key: "97619ced51c33688a79dd6db97ec9d30"
server_key_iv: "7b7a39655eb90f6d884a6c4601640054"
server_key_version: "encryptionkey@0"
I20260808 02:24:13.003839 21548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:24:13.007359 21566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:13.008896 21548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:13.009047 21548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "b6f4714804e04741b7164f282f9d039c"
format_stamp: "Formatted at 2026-08-08 02:24:12 on dist-test-slave-pgkr"
server_key: "97619ced51c33688a79dd6db97ec9d30"
server_key_iv: "7b7a39655eb90f6d884a6c4601640054"
server_key_version: "encryptionkey@0"
I20260808 02:24:13.009240 21548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:13.026098 21548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:13.031217 21548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:13.031481 21548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:13.055593 21548 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:34655
I20260808 02:24:13.055639 21630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:34655 every 8 connection(s)
I20260808 02:24:13.057116 21548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:13.061009 21631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:13.067340 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 21548
I20260808 02:24:13.067622 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:13.068486 20380 external_mini_cluster.cc:1428] Setting key bd4bb6c77be91ca28db7fcf1bdc6b71a
I20260808 02:24:13.073151 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c: Bootstrap starting.
I20260808 02:24:13.077270 21631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:13.078503 21631 log.cc:824] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:13.081833 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c: No bootstrap required, opened a new log
Aug 08 02:24:13 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:13.090706 21631 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f4714804e04741b7164f282f9d039c" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34655 } }
I20260808 02:24:13.091167 21631 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:13.091300 21631 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f4714804e04741b7164f282f9d039c, State: Initialized, Role: FOLLOWER
I20260808 02:24:13.092022 21631 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f4714804e04741b7164f282f9d039c" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34655 } }
I20260808 02:24:13.092332 21631 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:13.092489 21631 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:13.092693 21631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:13.093523 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:13.070932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.19.231.62:34655)
0808 02:24:13.071789 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:13.071801 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:13.071843 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:24:13.073692 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:13.073719 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:13.074130 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:13.074461 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:13.076535 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.077957 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.079724 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.080395 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.080538 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:13.085468 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:13.085476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:13.085494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:13.085539 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:13.088802 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:13.089524 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:13.089531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:13.089538 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:13.089715 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:13.090124 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:13.090128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:13.090130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:13.090735 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:13.091043 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:13.091519 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 02:24:13.091977 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":393,"threads_started":1}
I20260808 02:24:13.095443 21631 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f4714804e04741b7164f282f9d039c" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34655 } }
I20260808 02:24:13.096014 21631 leader_election.cc:302] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6f4714804e04741b7164f282f9d039c; no voters:
I20260808 02:24:13.096709 21631 leader_election.cc:288] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:13.096822 21636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:13.097550 21636 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [term 1 LEADER]: Becoming Leader. State: Replica: b6f4714804e04741b7164f282f9d039c, State: Running, Role: LEADER
I20260808 02:24:13.098160 21636 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f4714804e04741b7164f282f9d039c" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34655 } }
I20260808 02:24:13.099149 21631 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:13.101365 21637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6f4714804e04741b7164f282f9d039c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f4714804e04741b7164f282f9d039c" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34655 } } }
I20260808 02:24:13.101805 21637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:13.101945 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6f4714804e04741b7164f282f9d039c. Latest consensus state: current_term: 1 leader_uuid: "b6f4714804e04741b7164f282f9d039c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f4714804e04741b7164f282f9d039c" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34655 } } }
I20260808 02:24:13.102277 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:13.107322 21644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:13.112789 21644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:13.121763 21644 catalog_manager.cc:1347] Generated new cluster ID: d8c951a2e4c74c7782aa1510f5cf5828
I20260808 02:24:13.121886 21644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:13.139902 21644 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:13.141347 21644 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:13.149240 21644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c: Generated new TSK 0
I20260808 02:24:13.150056 21644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:13.199499 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:34655
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:13.544332 21658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:13.544636 21658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:13.544811 21658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:13.552500 21658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:13.552610 21658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:13.552789 21658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:24:13.552912 21658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:13.562935 21658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:34655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:13.564419 21658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:13.565802 21658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:13.579475 21668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:13.579509 21666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:13.579474 21665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:13.580549 21658 server_base.cc:1034] running on GCE node
I20260808 02:24:13.581351 21658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:13.582295 21658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:13.583542 21658 hybrid_clock.cc:648] HybridClock initialized: now 1786155853583429 us; error 88 us; skew 500 ppm
I20260808 02:24:13.583849 21658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:13 dist-test-slave-pgkr krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155853, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:13.587082 21658 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:13.588502 21658 webserver.cc:460] Webserver started at http://127.19.231.1:44641/ using document root <none> and password file <none>
I20260808 02:24:13.589238 21658 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:13.589327 21658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:13.589670 21658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:13.592669 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba"
format_stamp: "Formatted at 2026-08-08 02:24:13 on dist-test-slave-pgkr"
server_key: "2659751a67aed68c71a1250944097346"
server_key_iv: "793789243ed2ce317c230e85cd1ab0f7"
server_key_version: "encryptionkey@0"
I20260808 02:24:13.593467 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba"
format_stamp: "Formatted at 2026-08-08 02:24:13 on dist-test-slave-pgkr"
server_key: "2659751a67aed68c71a1250944097346"
server_key_iv: "793789243ed2ce317c230e85cd1ab0f7"
server_key_version: "encryptionkey@0"
I20260808 02:24:13.598850 21658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:24:13.602655 21675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:13.604038 21658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:13.604211 21658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba"
format_stamp: "Formatted at 2026-08-08 02:24:13 on dist-test-slave-pgkr"
server_key: "2659751a67aed68c71a1250944097346"
server_key_iv: "793789243ed2ce317c230e85cd1ab0f7"
server_key_version: "encryptionkey@0"
I20260808 02:24:13.604454 21658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:13.625286 21658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:13.629940 21658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:13.630231 21658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:13.633107 21658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:13.633220 21658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:13.633397 21658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:13.633481 21658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:24:13 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155853, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:13.658399 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:13.633911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:47705 (local address 127.19.231.62:34655)
0808 02:24:13.637345 (+ 3434us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:13.637350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:13.637371 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:24:13.639580 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:13.639589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:13.639690 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:13.639796 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:13.642428 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.643364 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.645964 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.646559 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.646760 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:13.652022 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:13.652027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:13.652030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:13.652063 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:13.654688 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:13.655741 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:13.655757 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:13.655764 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:13.655935 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:13.656569 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:13.656574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:13.656577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:13.656899 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:13.657053 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:13.657780 (+ 727us) server_negotiation.cc:299] Negotiation successful
0808 02:24:13.658010 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3319,"thread_start_us":3223,"threads_started":1}
I20260808 02:24:13.659830 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:13.634903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34655 (local address 127.19.231.1:47705)
0808 02:24:13.635995 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:13.636060 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:13.639144 (+ 3084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:13.640159 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:13.640175 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:13.641182 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:13.642208 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:13.642241 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.643558 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.643568 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:13.645806 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:13.645819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.646879 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.646890 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:13.647289 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:13.648205 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:13.648250 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:13.651779 (+ 3529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:13.654861 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:13.654875 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:13.654900 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:13.655589 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:13.656078 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:13.656083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:13.656091 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:13.656387 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:13.657044 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:13.657052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:13.657423 (+ 371us) client_negotiation.cc:769] Sending connection context
0808 02:24:13.657704 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 02:24:13.658268 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":428,"threads_started":1}
I20260808 02:24:13.685290 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:13.667714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34655 (local address 127.19.231.1:35205)
0808 02:24:13.668485 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:13.668510 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:13.668684 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:13.669015 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:13.669022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:13.669742 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:13.670228 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:13.670244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.671701 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.671704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:13.673092 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:13.673130 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.674556 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.674566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:13.674913 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:13.675792 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:13.675820 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:13.677359 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:13.681582 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:13.681589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:13.681596 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:13.682251 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:13.682748 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:13.682756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:13.682763 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:13.682970 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:13.683874 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:13.683882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:13.684166 (+ 284us) client_negotiation.cc:769] Sending connection context
0808 02:24:13.684311 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 02:24:13.684726 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":386,"threads_started":1}
I20260808 02:24:13.686553 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:13.667752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:35205 (local address 127.19.231.62:34655)
0808 02:24:13.668002 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:13.668012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:13.668606 (+ 594us) server_negotiation.cc:407] Connection header received
0808 02:24:13.668806 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:13.668810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:13.668872 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:13.668952 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:13.670435 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.671523 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.673287 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:13.674094 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:13.674244 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:13.677637 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:13.677647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:13.677655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:13.677705 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:13.681382 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:13.682435 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:13.682440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:13.682443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:13.682568 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:13.683174 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:13.683181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:13.683187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:13.683682 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:13.685619 (+ 1937us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:13.685803 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 02:24:13.686108 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:24:13.694219 21658 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:41735
I20260808 02:24:13.694280 21802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:41735 every 8 connection(s)
I20260808 02:24:13.695573 21658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:13.697655 21803 heartbeater.cc:347] Connected to a master server at 127.19.231.62:34655
I20260808 02:24:13.697898 21803 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:13.698546 21803 heartbeater.cc:511] Master 127.19.231.62:34655 requested a full tablet report, sending...
I20260808 02:24:13.700733 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 21658
I20260808 02:24:13.700780 21586 ts_manager.cc:194] Registered new tserver with Master: 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735)
I20260808 02:24:13.700974 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:13.701628 20380 external_mini_cluster.cc:1428] Setting key 0c735f304d84fca65b8b0f236e23596c
I20260808 02:24:13.702508 21586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:47705
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:13.750123 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:34655
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:14.089339 21807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:14.089680 21807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:14.089852 21807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:14.097445 21807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:14.097644 21807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:14.097800 21807 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:24:14.097951 21807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:14.108390 21807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:34655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:14.109869 21807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:14.111280 21807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:14.124629 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:14.124598 21813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:14.124543 21815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:14.126156 21807 server_base.cc:1034] running on GCE node
I20260808 02:24:14.126822 21807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:14.127707 21807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:14.128925 21807 hybrid_clock.cc:648] HybridClock initialized: now 1786155854128866 us; error 35 us; skew 500 ppm
I20260808 02:24:14.129235 21807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:14 dist-test-slave-pgkr krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:14.132442 21807 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:14.133917 21807 webserver.cc:460] Webserver started at http://127.19.231.2:36027/ using document root <none> and password file <none>
I20260808 02:24:14.134577 21807 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:14.134737 21807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:14.135016 21807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:14.137861 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "ded9a9a4750947b5b65fd3be9341cf8b"
format_stamp: "Formatted at 2026-08-08 02:24:14 on dist-test-slave-pgkr"
server_key: "366d405a65de5e272e1c109f3e82d427"
server_key_iv: "7a6a5df53293cba7ccdb62d2cee5ecc3"
server_key_version: "encryptionkey@0"
I20260808 02:24:14.138806 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "ded9a9a4750947b5b65fd3be9341cf8b"
format_stamp: "Formatted at 2026-08-08 02:24:14 on dist-test-slave-pgkr"
server_key: "366d405a65de5e272e1c109f3e82d427"
server_key_iv: "7a6a5df53293cba7ccdb62d2cee5ecc3"
server_key_version: "encryptionkey@0"
I20260808 02:24:14.143919 21807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:24:14.147405 21822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:14.148765 21807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:14.148929 21807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "ded9a9a4750947b5b65fd3be9341cf8b"
format_stamp: "Formatted at 2026-08-08 02:24:14 on dist-test-slave-pgkr"
server_key: "366d405a65de5e272e1c109f3e82d427"
server_key_iv: "7a6a5df53293cba7ccdb62d2cee5ecc3"
server_key_version: "encryptionkey@0"
I20260808 02:24:14.149138 21807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:14.165581 21807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:14.169536 21807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:14.169808 21807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:14.172590 21807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:14.172704 21807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:14.172881 21807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:14.172958 21807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:24:14 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:14.196754 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.172978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:54391 (local address 127.19.231.62:34655)
0808 02:24:14.173222 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.173232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.174721 (+ 1489us) server_negotiation.cc:407] Connection header received
0808 02:24:14.176836 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.176841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.176932 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.177040 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:14.179366 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.180748 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.183277 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.183922 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.184061 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.188840 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:14.188849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:14.188854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:14.188904 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:14.192439 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.193639 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.193645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.193658 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.193817 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.194423 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.194432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.194437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.194937 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:14.195139 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.195881 (+ 742us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.196209 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:24:14.197383 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.173640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34655 (local address 127.19.231.2:54391)
0808 02:24:14.174492 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:14.174537 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:14.176555 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:14.177331 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:14.177347 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:14.178202 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:14.179095 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.179129 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.180943 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.180950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.183111 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.183123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.184186 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.184194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.184407 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.185160 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:14.185201 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:14.188551 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:14.192598 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.192610 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.192634 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.193473 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.193930 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.193936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.193939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.194262 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.195087 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:14.195096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:14.195457 (+ 361us) client_negotiation.cc:769] Sending connection context
0808 02:24:14.195695 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 02:24:14.196105 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":375,"threads_started":1}
I20260808 02:24:14.220939 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.205215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:60317 (local address 127.19.231.62:34655)
0808 02:24:14.205463 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.205472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.205500 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:24:14.205620 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.205628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.205751 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.205886 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:14.206903 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.207778 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.209233 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.209833 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.209970 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.212388 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:14.212398 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:14.212405 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:14.212457 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:14.216775 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.217772 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.217778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.217781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.218024 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.218447 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.218452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.218455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.219149 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:14.219335 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.220178 (+ 843us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.220455 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:24:14.221586 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.204608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34655 (local address 127.19.231.2:60317)
0808 02:24:14.205340 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:14.205363 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:14.205500 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:14.205897 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:14.205900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:14.206331 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:14.206745 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.206757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.207903 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.207909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.209072 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.209085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.210015 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.210021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.210192 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.210968 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:14.210999 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:14.212144 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:14.216936 (+ 4792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.216951 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.216960 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.217623 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.218162 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.218166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.218168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.218319 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.219287 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:14.219291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:14.219709 (+ 418us) client_negotiation.cc:769] Sending connection context
0808 02:24:14.220042 (+ 333us) client_negotiation.cc:240] Negotiation successful
0808 02:24:14.220888 (+ 846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":655,"threads_started":1}
I20260808 02:24:14.241705 21807 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:42655
I20260808 02:24:14.241787 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:42655 every 8 connection(s)
I20260808 02:24:14.243330 21807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:14.246510 21948 heartbeater.cc:347] Connected to a master server at 127.19.231.62:34655
I20260808 02:24:14.246845 21948 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:14.247869 21948 heartbeater.cc:511] Master 127.19.231.62:34655 requested a full tablet report, sending...
I20260808 02:24:14.249505 21585 ts_manager.cc:194] Registered new tserver with Master: ded9a9a4750947b5b65fd3be9341cf8b (127.19.231.2:42655)
I20260808 02:24:14.250959 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 21807
I20260808 02:24:14.251108 21585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:54391
I20260808 02:24:14.251395 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:14.252281 20380 external_mini_cluster.cc:1428] Setting key 1c476a704ff4740d04363ab514a8fe0d
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:14.300141 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:34655
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:14.678047 21952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:14.678356 21952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:14.678462 21952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:14.686252 21952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:14.686383 21952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:14.686488 21952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:24:14.686573 21952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:14.698501 21952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:34655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:14.700150 21952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:14.701575 21952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:24:14.706029 21803 heartbeater.cc:503] Master 127.19.231.62:34655 was elected leader, sending a full tablet report...
W20260808 02:24:14.715015 21958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:14.715082 21960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:14.715106 21957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:14.715935 21952 server_base.cc:1034] running on GCE node
I20260808 02:24:14.716514 21952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:14.717396 21952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:14.718631 21952 hybrid_clock.cc:648] HybridClock initialized: now 1786155854718493 us; error 84 us; skew 500 ppm
I20260808 02:24:14.718976 21952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:14 dist-test-slave-pgkr krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:14.722227 21952 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:14.723634 21952 webserver.cc:460] Webserver started at http://127.19.231.3:46825/ using document root <none> and password file <none>
I20260808 02:24:14.724314 21952 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:14.724399 21952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:14.724723 21952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:14.727694 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "8ae1994a90704413b2310401dabc974d"
format_stamp: "Formatted at 2026-08-08 02:24:14 on dist-test-slave-pgkr"
server_key: "07c00824880a15da549cd630fb997c53"
server_key_iv: "bddba41dbd9c43d8480d36709338294a"
server_key_version: "encryptionkey@0"
I20260808 02:24:14.728432 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae1994a90704413b2310401dabc974d"
format_stamp: "Formatted at 2026-08-08 02:24:14 on dist-test-slave-pgkr"
server_key: "07c00824880a15da549cd630fb997c53"
server_key_iv: "bddba41dbd9c43d8480d36709338294a"
server_key_version: "encryptionkey@0"
I20260808 02:24:14.733455 21952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:24:14.736931 21967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:14.738333 21952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:14.738540 21952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "8ae1994a90704413b2310401dabc974d"
format_stamp: "Formatted at 2026-08-08 02:24:14 on dist-test-slave-pgkr"
server_key: "07c00824880a15da549cd630fb997c53"
server_key_iv: "bddba41dbd9c43d8480d36709338294a"
server_key_version: "encryptionkey@0"
I20260808 02:24:14.738843 21952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:14.751358 21952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:14.755755 21952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:14.756031 21952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:14.758690 21952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:14.758783 21952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:14.759009 21952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:14.759070 21952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:24:14 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:14.784520 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.759650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:55189 (local address 127.19.231.62:34655)
0808 02:24:14.760168 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.760176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.761680 (+ 1504us) server_negotiation.cc:407] Connection header received
0808 02:24:14.764540 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.764548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.764663 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.764807 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:14.767499 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.768649 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.770394 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.770955 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.771129 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.777015 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:14.777021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:14.777025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:14.777060 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:14.779988 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.781248 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.781256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.781262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.781459 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.782145 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.782151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.782156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.782440 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:14.782716 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.783926 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.784216 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260808 02:24:14.786448 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.760378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34655 (local address 127.19.231.3:55189)
0808 02:24:14.761439 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:14.761492 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:14.764162 (+ 2670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:14.765173 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:14.765200 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:14.766114 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:14.767233 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.767262 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.768828 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.768844 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.770215 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.770247 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.771291 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.771305 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.771625 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.772490 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:14.772528 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:14.776669 (+ 4141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:14.780178 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.780191 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.780213 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.781057 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.781643 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.781651 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.781657 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.781986 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.782700 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:14.782708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:14.783353 (+ 645us) client_negotiation.cc:769] Sending connection context
0808 02:24:14.783776 (+ 423us) client_negotiation.cc:240] Negotiation successful
0808 02:24:14.784518 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":458,"threads_started":1}
I20260808 02:24:14.808678 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.792752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:39873 (local address 127.19.231.62:34655)
0808 02:24:14.792935 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.792941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.793764 (+ 823us) server_negotiation.cc:407] Connection header received
0808 02:24:14.793969 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.793976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.794083 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.794197 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:14.796066 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.797358 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.798672 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.799451 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.799598 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.801497 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:14.801507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:14.801514 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:14.801565 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:14.805702 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.806438 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.806447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.806454 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.806669 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.807139 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.807144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.807148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.807500 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:14.807673 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.808202 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.808390 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:24:14.809021 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.792932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34655 (local address 127.19.231.3:39873)
0808 02:24:14.793644 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:14.793682 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:14.793837 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:14.794260 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:14.794268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:14.795138 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:14.795834 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.795852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.797486 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.797489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.798509 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.798521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.799632 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.799636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.799765 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.800515 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:14.800537 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:14.801282 (+ 745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:14.805866 (+ 4584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.805877 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.805882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.806276 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.806815 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:14.806823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:14.806827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:14.807013 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:14.807647 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:14.807653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:14.807877 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 02:24:14.808085 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 02:24:14.808464 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":446,"threads_started":1}
I20260808 02:24:14.819013 21952 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:46875
I20260808 02:24:14.819048 22096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:46875 every 8 connection(s)
I20260808 02:24:14.820248 21952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:14.821868 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 21952
I20260808 02:24:14.822096 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:14.822361 22097 heartbeater.cc:347] Connected to a master server at 127.19.231.62:34655
I20260808 02:24:14.822597 22097 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:14.822650 20380 external_mini_cluster.cc:1428] Setting key 2dea220ea2203ff07eb6fc1ad1b35679
I20260808 02:24:14.823313 22097 heartbeater.cc:511] Master 127.19.231.62:34655 requested a full tablet report, sending...
I20260808 02:24:14.824855 21585 ts_manager.cc:194] Registered new tserver with Master: 8ae1994a90704413b2310401dabc974d (127.19.231.3:46875)
I20260808 02:24:14.825735 21585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:55189
I20260808 02:24:14.837785 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:24:14 dist-test-slave-pgkr krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:24:14 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:14.890450 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.875847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.19.231.62:34655)
0808 02:24:14.876057 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.876062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.876360 (+ 298us) server_negotiation.cc:407] Connection header received
0808 02:24:14.876540 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.876545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.876623 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.876729 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:14.878217 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.879092 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.880386 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.880907 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.881078 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.885484 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:14.885489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:14.885493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:14.885530 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:14.888120 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.888768 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.888772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.888774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.888905 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:14.889256 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:14.889260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:14.889262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:14.889563 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:14.889794 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.890031 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.890202 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:24:14.894747 21585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:14.901875 21585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:14.932410 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.915042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:46875 (local address 127.0.0.1:55490)
0808 02:24:14.917573 (+ 2531us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:14.917599 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:14.917725 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:14.919181 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:14.919194 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:14.919240 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:14.919919 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.919940 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.922916 (+ 2976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.922926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.926339 (+ 3413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.926364 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.930880 (+ 4516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.930890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.931252 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.931338 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 02:24:14.931522 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 02:24:14.931855 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2315,"thread_start_us":2172,"threads_started":1}
I20260808 02:24:14.933557 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.915817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.19.231.3:46875)
0808 02:24:14.916994 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.917007 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.917884 (+ 877us) server_negotiation.cc:407] Connection header received
0808 02:24:14.918030 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.918043 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.918392 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.918703 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:14.920255 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.922675 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.926685 (+ 4010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.928914 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.929115 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.929477 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.933009 (+ 3532us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.933209 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":759,"threads_started":1}
I20260808 02:24:14.939028 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 96b49e61bb544c2880db1082c816daff (DEFAULT_TABLE table=test-table [id=bc163b7437634a848c1c9c26d46a504a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:14.939741 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.914771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:41735 (local address 127.0.0.1:52230)
0808 02:24:14.915892 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:14.915948 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:14.916171 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:14.931076 (+ 14905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:14.931086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:14.931131 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:14.931820 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.931838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.934842 (+ 3004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.934849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.936838 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.936858 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.938826 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.938833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.939022 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.939067 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:24:14.939183 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 02:24:14.939326 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":666,"threads_started":1}
I20260808 02:24:14.940702 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.914999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.19.231.1:41735)
0808 02:24:14.930121 (+ 15122us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.930136 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.930179 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:24:14.930320 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.930333 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.930743 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.932109 (+ 1366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:14.932442 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.934581 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.937040 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.938558 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.938750 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.939055 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.940135 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.940348 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14867,"thread_start_us":14666,"threads_started":1}
I20260808 02:24:14.940796 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b49e61bb544c2880db1082c816daff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:14.941402 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.917671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:42655 (local address 127.0.0.1:37282)
0808 02:24:14.932688 (+ 15017us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:14.932707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:14.932832 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:14.934982 (+ 2150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:14.934991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:14.935018 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:14.935167 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.935175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.936751 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.936756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.938560 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:14.938574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.940698 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.940704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:14.940898 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.940927 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:24:14.941009 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:24:14.941145 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14918,"thread_start_us":17134,"threads_started":1}
I20260808 02:24:14.941780 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:14.917982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.19.231.2:42655)
0808 02:24:14.934214 (+ 16232us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:14.934223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:14.934252 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:24:14.934347 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:14.934352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:14.934544 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:14.934795 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:14.935449 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.936597 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.938782 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:14.940519 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:14.940672 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:14.941077 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:14.941142 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 02:24:14.941313 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16057,"thread_start_us":16083,"threads_started":1}
I20260808 02:24:14.944558 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 96b49e61bb544c2880db1082c816daff (DEFAULT_TABLE table=test-table [id=bc163b7437634a848c1c9c26d46a504a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:14.946058 21736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b49e61bb544c2880db1082c816daff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:14.948167 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 96b49e61bb544c2880db1082c816daff (DEFAULT_TABLE table=test-table [id=bc163b7437634a848c1c9c26d46a504a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:14.949349 21873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b49e61bb544c2880db1082c816daff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:14.956995 22117 tablet_bootstrap.cc:492] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Bootstrap starting.
I20260808 02:24:14.960870 22117 tablet_bootstrap.cc:654] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:14.961019 22118 tablet_bootstrap.cc:492] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Bootstrap starting.
I20260808 02:24:14.962628 22119 tablet_bootstrap.cc:492] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Bootstrap starting.
I20260808 02:24:14.962699 22117 log.cc:824] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:14.965123 22118 tablet_bootstrap.cc:654] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:14.966302 22119 tablet_bootstrap.cc:654] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:14.966521 22117 tablet_bootstrap.cc:492] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: No bootstrap required, opened a new log
I20260808 02:24:14.966720 22118 log.cc:824] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:14.966825 22117 ts_tablet_manager.cc:1397] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260808 02:24:14.967638 22119 log.cc:824] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:14.970326 22118 tablet_bootstrap.cc:492] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: No bootstrap required, opened a new log
I20260808 02:24:14.970715 22118 ts_tablet_manager.cc:1397] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 02:24:14.973332 22119 tablet_bootstrap.cc:492] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: No bootstrap required, opened a new log
I20260808 02:24:14.973619 22119 ts_tablet_manager.cc:1397] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260808 02:24:14.974888 22117 raft_consensus.cc:348] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:14.975260 22117 raft_consensus.cc:374] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:14.975419 22117 raft_consensus.cc:729] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1994a90704413b2310401dabc974d, State: Initialized, Role: FOLLOWER
I20260808 02:24:14.976001 22117 consensus_queue.cc:260] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:14.977135 22097 heartbeater.cc:503] Master 127.19.231.62:34655 was elected leader, sending a full tablet report...
I20260808 02:24:14.977886 22117 ts_tablet_manager.cc:1428] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260808 02:24:14.982466 22118 raft_consensus.cc:348] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:14.982944 22118 raft_consensus.cc:374] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:14.983060 22118 raft_consensus.cc:729] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eba9a494fc24e43be6a76f3ebc7d5ba, State: Initialized, Role: FOLLOWER
I20260808 02:24:14.983642 22118 consensus_queue.cc:260] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:14.983896 22119 raft_consensus.cc:348] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:14.984371 22119 raft_consensus.cc:374] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:14.984481 22119 raft_consensus.cc:729] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded9a9a4750947b5b65fd3be9341cf8b, State: Initialized, Role: FOLLOWER
I20260808 02:24:14.985059 22119 consensus_queue.cc:260] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:14.985299 22118 ts_tablet_manager.cc:1428] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260808 02:24:14.986090 21948 heartbeater.cc:503] Master 127.19.231.62:34655 was elected leader, sending a full tablet report...
I20260808 02:24:14.986891 22119 ts_tablet_manager.cc:1428] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Time spent starting tablet: real 0.013s user 0.014s sys 0.002s
W20260808 02:24:14.996585 21949 tablet.cc:2369] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:24:15.071890 22098 tablet.cc:2369] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:24:15.198844 21804 tablet.cc:2369] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:15.259438 22124 raft_consensus.cc:482] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:15.259745 22124 raft_consensus.cc:504] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:15.261405 22124 leader_election.cc:288] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735), ded9a9a4750947b5b65fd3be9341cf8b (127.19.231.2:42655)
I20260808 02:24:15.267207 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.261809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:41735 (local address 127.19.231.3:59909)
0808 02:24:15.262019 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:15.262040 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:15.262170 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:15.262511 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:15.262516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:15.262550 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:15.262786 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.262795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.263839 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.263844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.265317 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.265332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.266377 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.266383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.266559 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.266582 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:24:15.266738 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:24:15.266874 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 02:24:15.267220 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.261811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:59909 (local address 127.19.231.1:41735)
0808 02:24:15.262215 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:15.262220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:15.262238 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:24:15.262312 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:15.262316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:15.262374 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:15.262482 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:15.262933 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.263699 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.265474 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.266227 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.266392 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.266469 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:15.266799 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 02:24:15.266964 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260808 02:24:15.267946 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.262085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:42655 (local address 127.19.231.3:51629)
0808 02:24:15.262760 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:15.262809 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:15.262957 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:15.263260 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:15.263264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:15.263288 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:15.263603 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.263611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.264666 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.264672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.265961 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.265972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.267295 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.267304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.267485 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.267530 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:24:15.267600 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 02:24:15.267684 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":306,"threads_started":1}
I20260808 02:24:15.268023 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.261949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:51629 (local address 127.19.231.2:42655)
0808 02:24:15.262145 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:15.262151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:15.262900 (+ 749us) server_negotiation.cc:407] Connection header received
0808 02:24:15.263089 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:15.263093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:15.263154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:15.263246 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:15.263756 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.264522 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.266113 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.267062 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.267231 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.267314 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:15.267651 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 02:24:15.267783 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:24:15.268455 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b49e61bb544c2880db1082c816daff" candidate_uuid: "8ae1994a90704413b2310401dabc974d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" is_pre_election: true
I20260808 02:24:15.269032 21757 raft_consensus.cc:2454] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1994a90704413b2310401dabc974d in term 0.
I20260808 02:24:15.269044 21893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b49e61bb544c2880db1082c816daff" candidate_uuid: "8ae1994a90704413b2310401dabc974d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" is_pre_election: true
I20260808 02:24:15.269603 21893 raft_consensus.cc:2454] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1994a90704413b2310401dabc974d in term 0.
I20260808 02:24:15.269682 21968 leader_election.cc:302] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eba9a494fc24e43be6a76f3ebc7d5ba, 8ae1994a90704413b2310401dabc974d; no voters:
I20260808 02:24:15.270097 22124 raft_consensus.cc:2781] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:15.270258 22124 raft_consensus.cc:482] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:15.270373 22124 raft_consensus.cc:3037] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:15.272661 22124 raft_consensus.cc:504] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:15.273341 22124 leader_election.cc:288] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [CANDIDATE]: Term 1 election: Requested vote from peers 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735), ded9a9a4750947b5b65fd3be9341cf8b (127.19.231.2:42655)
I20260808 02:24:15.273723 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b49e61bb544c2880db1082c816daff" candidate_uuid: "8ae1994a90704413b2310401dabc974d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba"
I20260808 02:24:15.273777 21893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b49e61bb544c2880db1082c816daff" candidate_uuid: "8ae1994a90704413b2310401dabc974d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded9a9a4750947b5b65fd3be9341cf8b"
I20260808 02:24:15.273962 21757 raft_consensus.cc:3037] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:15.273962 21893 raft_consensus.cc:3037] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:15.277247 21893 raft_consensus.cc:2454] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1994a90704413b2310401dabc974d in term 1.
I20260808 02:24:15.277251 21757 raft_consensus.cc:2454] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1994a90704413b2310401dabc974d in term 1.
I20260808 02:24:15.277709 21968 leader_election.cc:302] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0eba9a494fc24e43be6a76f3ebc7d5ba, 8ae1994a90704413b2310401dabc974d, ded9a9a4750947b5b65fd3be9341cf8b; no voters:
I20260808 02:24:15.278057 22124 raft_consensus.cc:2781] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:15.278514 22124 raft_consensus.cc:686] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1994a90704413b2310401dabc974d, State: Running, Role: LEADER
I20260808 02:24:15.279014 22124 consensus_queue.cc:237] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } }
I20260808 02:24:15.283789 21586 catalog_manager.cc:5495] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1994a90704413b2310401dabc974d (127.19.231.3). New cstate: current_term: 1 leader_uuid: "8ae1994a90704413b2310401dabc974d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20260808 02:24:15.389204 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.383415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.19.231.3:46875)
0808 02:24:15.383640 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:15.383648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:15.383841 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:24:15.384205 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:15.384210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:15.384299 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:15.384443 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:15.384823 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.385578 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.386944 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.387347 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.387481 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.387978 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:15.388490 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:15.388679 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:15.388827 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 02:24:15.388910 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:24:15.396813 21757 raft_consensus.cc:1261] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Refusing update from remote peer 8ae1994a90704413b2310401dabc974d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:15.396806 21893 raft_consensus.cc:1261] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1994a90704413b2310401dabc974d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:15.397773 22128 consensus_queue.cc:1035] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:15.398115 22124 consensus_queue.cc:1035] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:15.410179 22131 mvcc.cc:205] Tried to move back new op lower bound from 7316094383695654912 to 7316094383226642432. Current Snapshot: MvccSnapshot[applied={T|T < 7316094383695654912}]
I20260808 02:24:15.429658 21586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:15.429939 21586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:15.433995 21586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c: Sending DeleteTablet for 3 replicas of tablet 96b49e61bb544c2880db1082c816daff
I20260808 02:24:15.435374 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b49e61bb544c2880db1082c816daff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:15 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:52230
I20260808 02:24:15.435501 21873 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b49e61bb544c2880db1082c816daff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:15 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:37282
I20260808 02:24:15.435523 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b49e61bb544c2880db1082c816daff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:15 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:55490
I20260808 02:24:15.440953 21586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:15.441957 21586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:15.450760 22021 tablet_service.cc:1467] Processing CreateTablet for tablet efb2e7047913495fa08be86806786d5d (DEFAULT_TABLE table=test-table [id=2f629cfef7ac4b22974e861db3aaeeed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:15.450907 21872 tablet_service.cc:1467] Processing CreateTablet for tablet efb2e7047913495fa08be86806786d5d (DEFAULT_TABLE table=test-table [id=2f629cfef7ac4b22974e861db3aaeeed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:15.451517 22021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb2e7047913495fa08be86806786d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:15.451562 21872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb2e7047913495fa08be86806786d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:15.456277 21735 tablet_service.cc:1467] Processing CreateTablet for tablet efb2e7047913495fa08be86806786d5d (DEFAULT_TABLE table=test-table [id=2f629cfef7ac4b22974e861db3aaeeed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:15.457196 21735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb2e7047913495fa08be86806786d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:15.459944 22119 tablet_bootstrap.cc:492] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: Bootstrap starting.
I20260808 02:24:15.461205 22117 tablet_bootstrap.cc:492] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: Bootstrap starting.
I20260808 02:24:15.462657 22119 tablet_bootstrap.cc:654] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:15.464035 22117 tablet_bootstrap.cc:654] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:15.472934 22144 tablet_replica.cc:330] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: stopping tablet replica
I20260808 02:24:15.473716 22144 raft_consensus.cc:2229] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:15.475370 22144 raft_consensus.cc:2258] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:15.475911 22142 tablet_replica.cc:330] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: stopping tablet replica
I20260808 02:24:15.474259 22119 tablet_bootstrap.cc:492] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: No bootstrap required, opened a new log
I20260808 02:24:15.476302 22119 ts_tablet_manager.cc:1397] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20260808 02:24:15.477834 22119 raft_consensus.cc:348] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.478330 22119 raft_consensus.cc:374] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:15.478554 22119 raft_consensus.cc:729] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded9a9a4750947b5b65fd3be9341cf8b, State: Initialized, Role: FOLLOWER
I20260808 02:24:15.479153 22142 raft_consensus.cc:2229] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:15.479192 22119 consensus_queue.cc:260] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.479627 22118 tablet_bootstrap.cc:492] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: Bootstrap starting.
I20260808 02:24:15.480059 22142 raft_consensus.cc:2258] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:15.482640 22118 tablet_bootstrap.cc:654] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:15.482726 22144 ts_tablet_manager.cc:1905] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:15.485581 22142 ts_tablet_manager.cc:1905] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:15.487174 22119 ts_tablet_manager.cc:1428] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260808 02:24:15.487210 22143 tablet_replica.cc:330] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: stopping tablet replica
I20260808 02:24:15.487715 22117 tablet_bootstrap.cc:492] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: No bootstrap required, opened a new log
I20260808 02:24:15.487823 22143 raft_consensus.cc:2229] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:15.487990 22117 ts_tablet_manager.cc:1397] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20260808 02:24:15.488539 22118 tablet_bootstrap.cc:492] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: No bootstrap required, opened a new log
I20260808 02:24:15.488775 22118 ts_tablet_manager.cc:1397] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260808 02:24:15.490250 22143 raft_consensus.cc:2258] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:15.490660 22117 raft_consensus.cc:348] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.490949 22117 raft_consensus.cc:374] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:15.491187 22117 raft_consensus.cc:729] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1994a90704413b2310401dabc974d, State: Initialized, Role: FOLLOWER
I20260808 02:24:15.491669 22117 consensus_queue.cc:260] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.492967 22143 ts_tablet_manager.cc:1905] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:15.495674 22118 raft_consensus.cc:348] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.496169 22118 raft_consensus.cc:374] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:15.495391 22117 ts_tablet_manager.cc:1428] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260808 02:24:15.499132 22118 raft_consensus.cc:729] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eba9a494fc24e43be6a76f3ebc7d5ba, State: Initialized, Role: FOLLOWER
I20260808 02:24:15.500555 22118 consensus_queue.cc:260] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.502110 22144 ts_tablet_manager.cc:1918] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:15.502462 22144 log.cc:1196] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/96b49e61bb544c2880db1082c816daff
I20260808 02:24:15.503620 22144 ts_tablet_manager.cc:1939] T 96b49e61bb544c2880db1082c816daff P ded9a9a4750947b5b65fd3be9341cf8b: Deleting consensus metadata
I20260808 02:24:15.504812 22118 ts_tablet_manager.cc:1428] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: Time spent starting tablet: real 0.016s user 0.002s sys 0.002s
I20260808 02:24:15.506862 21568 catalog_manager.cc:4841] TS ded9a9a4750947b5b65fd3be9341cf8b (127.19.231.2:42655): tablet 96b49e61bb544c2880db1082c816daff (table test-table [id=bc163b7437634a848c1c9c26d46a504a]) successfully deleted
I20260808 02:24:15.509750 22143 ts_tablet_manager.cc:1918] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:15.510088 22143 log.cc:1196] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/96b49e61bb544c2880db1082c816daff
I20260808 02:24:15.511238 22143 ts_tablet_manager.cc:1939] T 96b49e61bb544c2880db1082c816daff P 8ae1994a90704413b2310401dabc974d: Deleting consensus metadata
I20260808 02:24:15.513792 21568 catalog_manager.cc:4841] TS 8ae1994a90704413b2310401dabc974d (127.19.231.3:46875): tablet 96b49e61bb544c2880db1082c816daff (table test-table [id=bc163b7437634a848c1c9c26d46a504a]) successfully deleted
I20260808 02:24:15.514478 22142 ts_tablet_manager.cc:1918] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:15.514786 22142 log.cc:1196] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/96b49e61bb544c2880db1082c816daff
I20260808 02:24:15.515627 22142 ts_tablet_manager.cc:1939] T 96b49e61bb544c2880db1082c816daff P 0eba9a494fc24e43be6a76f3ebc7d5ba: Deleting consensus metadata
I20260808 02:24:15.517597 21567 catalog_manager.cc:4841] TS 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735): tablet 96b49e61bb544c2880db1082c816daff (table test-table [id=bc163b7437634a848c1c9c26d46a504a]) successfully deleted
I20260808 02:24:15.674820 22126 raft_consensus.cc:482] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:15.675205 22126 raft_consensus.cc:504] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.679943 22126 leader_election.cc:288] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735), 8ae1994a90704413b2310401dabc974d (127.19.231.3:46875)
I20260808 02:24:15.698925 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.678875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:46563 (local address 127.19.231.1:41735)
0808 02:24:15.679108 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:15.679114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:15.685010 (+ 5896us) server_negotiation.cc:407] Connection header received
0808 02:24:15.685826 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:15.685835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:15.685927 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:15.685992 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:15.688035 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.689728 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.694596 (+ 4868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.696704 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.696917 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.696999 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:15.698159 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0808 02:24:15.698367 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 02:24:15.699975 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.678872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:40007 (local address 127.19.231.3:46875)
0808 02:24:15.679116 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:15.679123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:15.685717 (+ 6594us) server_negotiation.cc:407] Connection header received
0808 02:24:15.685962 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:15.685969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:15.686251 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:15.686323 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:15.688234 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.689768 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.694549 (+ 4781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.696693 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.696848 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.696945 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:15.699174 (+ 2229us) server_negotiation.cc:299] Negotiation successful
0808 02:24:15.699329 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 02:24:15.700320 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.678731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:41735 (local address 127.19.231.2:46563)
0808 02:24:15.684806 (+ 6075us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:15.684842 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:15.685609 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:15.686369 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:15.686374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:15.686400 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:15.687776 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.687794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.689946 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.689952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.694382 (+ 4430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.694397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.697494 (+ 3097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.697500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.697855 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.697912 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 02:24:15.699305 (+ 1393us) client_negotiation.cc:240] Negotiation successful
0808 02:24:15.699458 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5807,"mutex_wait_us":148,"thread_start_us":5524,"threads_started":1}
I20260808 02:24:15.701310 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.679110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:46875 (local address 127.19.231.2:40007)
0808 02:24:15.685602 (+ 6492us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:15.685620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:15.685809 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:15.686731 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:15.686735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:15.686781 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:15.687964 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.687978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.689946 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.689952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.694285 (+ 4333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.694312 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.697395 (+ 3083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.697403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.698971 (+ 1568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.699007 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:24:15.700234 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0808 02:24:15.700397 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6255,"mutex_wait_us":1150,"thread_start_us":5941,"threads_started":1}
I20260808 02:24:15.702630 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb2e7047913495fa08be86806786d5d" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" is_pre_election: true
I20260808 02:24:15.703034 21757 raft_consensus.cc:2454] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 0.
I20260808 02:24:15.702991 22049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb2e7047913495fa08be86806786d5d" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1994a90704413b2310401dabc974d" is_pre_election: true
I20260808 02:24:15.703972 22049 raft_consensus.cc:2454] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 0.
I20260808 02:24:15.705056 21823 leader_election.cc:302] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eba9a494fc24e43be6a76f3ebc7d5ba, ded9a9a4750947b5b65fd3be9341cf8b; no voters:
I20260808 02:24:15.705581 22126 raft_consensus.cc:2781] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:15.705752 22126 raft_consensus.cc:482] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:15.705968 22126 raft_consensus.cc:3037] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:15.710990 22126 raft_consensus.cc:504] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.713188 22126 leader_election.cc:288] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 election: Requested vote from peers 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735), 8ae1994a90704413b2310401dabc974d (127.19.231.3:46875)
I20260808 02:24:15.713169 22049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb2e7047913495fa08be86806786d5d" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1994a90704413b2310401dabc974d"
I20260808 02:24:15.713248 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb2e7047913495fa08be86806786d5d" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba"
I20260808 02:24:15.713474 22049 raft_consensus.cc:3037] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:15.713629 21757 raft_consensus.cc:3037] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:15.718397 22049 raft_consensus.cc:2454] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 1.
I20260808 02:24:15.718415 21757 raft_consensus.cc:2454] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 1.
I20260808 02:24:15.719112 21824 leader_election.cc:302] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1994a90704413b2310401dabc974d, ded9a9a4750947b5b65fd3be9341cf8b; no voters:
I20260808 02:24:15.719763 22126 raft_consensus.cc:2781] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:15.720842 22126 raft_consensus.cc:686] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 1 LEADER]: Becoming Leader. State: Replica: ded9a9a4750947b5b65fd3be9341cf8b, State: Running, Role: LEADER
I20260808 02:24:15.721493 22126 consensus_queue.cc:237] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.726588 21586 catalog_manager.cc:5495] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b reported cstate change: term changed from 0 to 1, leader changed from <none> to ded9a9a4750947b5b65fd3be9341cf8b (127.19.231.2). New cstate: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:24:15 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155852, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:15.844708 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.814222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34655 (local address 127.0.0.1:59120)
0808 02:24:15.815985 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:15.816008 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:15.816241 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:15.816967 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:15.816973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:15.818775 (+ 1802us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:15.819533 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.819559 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.822063 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.822077 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.824738 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:15.824755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.826154 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.826165 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:15.826498 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.827548 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:15.827575 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:15.832763 (+ 5188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:15.839949 (+ 7186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:15.839966 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:15.839975 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:15.841082 (+ 1107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:15.841829 (+ 747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:15.841835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:15.841840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:15.842118 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:15.843012 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:15.843022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:15.843420 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 02:24:15.843638 (+ 218us) client_negotiation.cc:240] Negotiation successful
0808 02:24:15.843963 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":1079,"threads_started":1}
I20260808 02:24:15.845396 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:15.814080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.19.231.62:34655)
0808 02:24:15.816589 (+ 2509us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:15.816593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:15.816628 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:24:15.816678 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:15.816682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:15.816738 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:15.816832 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:15.819763 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.821864 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.824917 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:15.825824 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:15.825949 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:15.833254 (+ 7305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:15.833271 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:15.833279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:15.833329 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:15.839758 (+ 6429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:15.841273 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:15.841280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:15.841283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:15.841613 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:15.842325 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:15.842341 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:15.842345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:15.842870 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:15.843189 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:15.843844 (+ 655us) server_negotiation.cc:299] Negotiation successful
0808 02:24:15.844131 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2361,"thread_start_us":2101,"threads_started":1}
I20260808 02:24:15.854352 21586 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:59120:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:24:15.855940 21586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:15.864702 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 489a3b625ad942b68829e814b302eda1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03688d8ce3904b0ab84ddf2803ddec19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:24:15.865206 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 489a3b625ad942b68829e814b302eda1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03688d8ce3904b0ab84ddf2803ddec19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:24:15.865482 21735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489a3b625ad942b68829e814b302eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:15.866055 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489a3b625ad942b68829e814b302eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:15.868909 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 489a3b625ad942b68829e814b302eda1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03688d8ce3904b0ab84ddf2803ddec19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:24:15.869937 21873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489a3b625ad942b68829e814b302eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:15.877336 22118 tablet_bootstrap.cc:492] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: Bootstrap starting.
I20260808 02:24:15.879995 22118 tablet_bootstrap.cc:654] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:15.885942 22119 tablet_bootstrap.cc:492] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: Bootstrap starting.
I20260808 02:24:15.887137 22118 tablet_bootstrap.cc:492] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: No bootstrap required, opened a new log
I20260808 02:24:15.887385 22118 ts_tablet_manager.cc:1397] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260808 02:24:15.888196 22117 tablet_bootstrap.cc:492] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: Bootstrap starting.
I20260808 02:24:15.888266 22118 raft_consensus.cc:348] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.888526 22118 raft_consensus.cc:374] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:15.888612 22118 raft_consensus.cc:729] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eba9a494fc24e43be6a76f3ebc7d5ba, State: Initialized, Role: FOLLOWER
I20260808 02:24:15.888887 22118 consensus_queue.cc:260] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.889135 22119 tablet_bootstrap.cc:654] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:15.890164 22125 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:15.890861 22125 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:15.891798 22118 ts_tablet_manager.cc:1428] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 02:24:15.893146 22117 tablet_bootstrap.cc:654] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:15.897715 22119 tablet_bootstrap.cc:492] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: No bootstrap required, opened a new log
I20260808 02:24:15.897964 22119 ts_tablet_manager.cc:1397] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20260808 02:24:15.899317 22119 raft_consensus.cc:348] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.899870 22119 raft_consensus.cc:374] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:15.900060 22119 raft_consensus.cc:729] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded9a9a4750947b5b65fd3be9341cf8b, State: Initialized, Role: FOLLOWER
I20260808 02:24:15.900697 22119 consensus_queue.cc:260] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.901304 22126 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:15.901722 22126 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:15.902576 22119 ts_tablet_manager.cc:1428] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 02:24:15.904217 22117 tablet_bootstrap.cc:492] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: No bootstrap required, opened a new log
I20260808 02:24:15.904944 22117 ts_tablet_manager.cc:1397] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260808 02:24:15.906206 22117 raft_consensus.cc:348] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.906662 22117 raft_consensus.cc:374] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:15.906771 22117 raft_consensus.cc:729] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1994a90704413b2310401dabc974d, State: Initialized, Role: FOLLOWER
I20260808 02:24:15.907210 22117 consensus_queue.cc:260] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:15.908527 22137 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:15.909453 22137 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:15.910049 22117 ts_tablet_manager.cc:1428] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260808 02:24:16.036069 22126 raft_consensus.cc:482] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:16.036306 22126 raft_consensus.cc:504] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:16.036981 22126 leader_election.cc:288] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735), 8ae1994a90704413b2310401dabc974d (127.19.231.3:46875)
I20260808 02:24:16.037639 22049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489a3b625ad942b68829e814b302eda1" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1994a90704413b2310401dabc974d" is_pre_election: true
I20260808 02:24:16.037900 22049 raft_consensus.cc:2454] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 0.
I20260808 02:24:16.037782 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489a3b625ad942b68829e814b302eda1" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" is_pre_election: true
I20260808 02:24:16.038343 21824 leader_election.cc:302] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1994a90704413b2310401dabc974d, ded9a9a4750947b5b65fd3be9341cf8b; no voters:
I20260808 02:24:16.038338 21757 raft_consensus.cc:2454] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 0.
I20260808 02:24:16.038651 22126 raft_consensus.cc:2781] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:16.038780 22126 raft_consensus.cc:482] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:16.038906 22126 raft_consensus.cc:3037] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:16.041247 22126 raft_consensus.cc:504] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:16.041944 22126 leader_election.cc:288] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 election: Requested vote from peers 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735), 8ae1994a90704413b2310401dabc974d (127.19.231.3:46875)
I20260808 02:24:16.042461 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489a3b625ad942b68829e814b302eda1" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba"
I20260808 02:24:16.042531 22049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489a3b625ad942b68829e814b302eda1" candidate_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1994a90704413b2310401dabc974d"
I20260808 02:24:16.042786 21757 raft_consensus.cc:3037] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:16.042904 22049 raft_consensus.cc:3037] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:16.046669 21757 raft_consensus.cc:2454] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 1.
I20260808 02:24:16.047236 21823 leader_election.cc:302] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eba9a494fc24e43be6a76f3ebc7d5ba, ded9a9a4750947b5b65fd3be9341cf8b; no voters:
I20260808 02:24:16.047254 22049 raft_consensus.cc:2454] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9a9a4750947b5b65fd3be9341cf8b in term 1.
I20260808 02:24:16.047623 22126 raft_consensus.cc:2781] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:16.047922 22126 raft_consensus.cc:686] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [term 1 LEADER]: Becoming Leader. State: Replica: ded9a9a4750947b5b65fd3be9341cf8b, State: Running, Role: LEADER
I20260808 02:24:16.048288 22126 consensus_queue.cc:237] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } }
I20260808 02:24:16.050808 22150 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: TxnStatusTablet state changed. Reason: New leader ded9a9a4750947b5b65fd3be9341cf8b. Latest consensus state: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:16.051395 22150 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:24:16.052814 22163 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:24:16.053210 22163 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:24:16.052942 21585 catalog_manager.cc:5495] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b reported cstate change: term changed from 0 to 1, leader changed from <none> to ded9a9a4750947b5b65fd3be9341cf8b (127.19.231.2). New cstate: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20260808 02:24:16.106066 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.092403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.19.231.2:42655)
0808 02:24:16.093647 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.093653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.093679 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:24:16.094137 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.094145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.094288 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.095381 (+ 1093us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:16.096009 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.098043 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.101273 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.102399 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.103355 (+ 956us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.103561 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:16.104805 (+ 1244us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:16.105039 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.105206 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.105295 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":789,"threads_started":1}
I20260808 02:24:16.106096 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.092755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:42655 (local address 127.0.0.1:50720)
0808 02:24:16.093544 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:16.093580 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:16.093830 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:16.094651 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:16.094679 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:16.094708 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:24:16.095218 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:16.095234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.098293 (+ 3059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.098299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:16.100944 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:16.100961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.102587 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.102594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:16.103020 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.103132 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:24:16.104998 (+ 1866us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:24:16.105029 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:24:16.105157 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:24:16.105481 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467}
I20260808 02:24:16.115068 21757 raft_consensus.cc:1261] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Refusing update from remote peer ded9a9a4750947b5b65fd3be9341cf8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:16.115094 22049 raft_consensus.cc:1261] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Refusing update from remote peer ded9a9a4750947b5b65fd3be9341cf8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:16.116114 22150 consensus_queue.cc:1035] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:16.117034 22150 consensus_queue.cc:1035] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:16.135757 22125 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: TxnStatusTablet state changed. Reason: New leader ded9a9a4750947b5b65fd3be9341cf8b. Latest consensus state: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:16.136499 22125 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:16.144881 22169 mvcc.cc:205] Tried to move back new op lower bound from 7316094386633797632 to 7316094386376110080. Current Snapshot: MvccSnapshot[applied={T|T < 7316094386633797632}]
I20260808 02:24:16.148428 22125 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:16.148887 22125 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P 0eba9a494fc24e43be6a76f3ebc7d5ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:16.150132 22137 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: TxnStatusTablet state changed. Reason: New leader ded9a9a4750947b5b65fd3be9341cf8b. Latest consensus state: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:16.150715 22137 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:16.161921 22150 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:16.162453 22150 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:24:16.173141 22150 consensus_queue.cc:1035] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 02:24:16.176604 22137 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:16.177066 22137 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P 8ae1994a90704413b2310401dabc974d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:16.214771 22150 consensus_queue.cc:1035] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:16.237880 22150 tablet_replica.cc:439] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eba9a494fc24e43be6a76f3ebc7d5ba" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41735 } } peers { permanent_uuid: "ded9a9a4750947b5b65fd3be9341cf8b" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42655 } } peers { permanent_uuid: "8ae1994a90704413b2310401dabc974d" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46875 } } }
I20260808 02:24:16.238318 22150 tablet_replica.cc:442] T 489a3b625ad942b68829e814b302eda1 P ded9a9a4750947b5b65fd3be9341cf8b: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:24:16.253647 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.194250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.19.231.2:42655)
0808 02:24:16.194483 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.194492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.230480 (+ 35988us) server_negotiation.cc:407] Connection header received
0808 02:24:16.238263 (+ 7783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.238286 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.238397 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.238473 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:16.245815 (+ 7342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.247578 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.250761 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.251721 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.251892 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.252688 (+ 796us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:16.252822 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:16.252895 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.253158 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.253244 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:24:16.305001 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.293577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:42655 (local address 127.19.231.2:35167)
0808 02:24:16.295586 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:16.295603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:16.295757 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:16.296209 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:16.296216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:16.296226 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:24:16.297051 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:16.297062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.299409 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.299413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:16.301860 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:16.301875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.303309 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.303315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:16.303494 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.303558 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:24:16.304036 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:24:16.304085 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:24:16.304165 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 02:24:16.304270 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1750,"mutex_wait_us":2,"thread_start_us":1416,"threads_started":1}
I20260808 02:24:16.305090 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.294295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:35167 (local address 127.19.231.2:42655)
0808 02:24:16.294686 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.294696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.295700 (+ 1004us) server_negotiation.cc:407] Connection header received
0808 02:24:16.295884 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.295890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.296060 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.296208 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:16.297296 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.299205 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.302033 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.303095 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.303246 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.303781 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:16.303922 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:16.304036 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.304320 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.304406 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":63}
I20260808 02:24:16.384740 21586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:16.385020 21586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:16.388114 21586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6f4714804e04741b7164f282f9d039c: Sending DeleteTablet for 3 replicas of tablet efb2e7047913495fa08be86806786d5d
I20260808 02:24:16.389094 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet efb2e7047913495fa08be86806786d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:16 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:52230
I20260808 02:24:16.389318 21873 tablet_service.cc:1514] Processing DeleteTablet for tablet efb2e7047913495fa08be86806786d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:16 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:37282
I20260808 02:24:16.389535 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet efb2e7047913495fa08be86806786d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:16 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:55490
I20260808 02:24:16.401062 22186 tablet_replica.cc:330] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: stopping tablet replica
I20260808 02:24:16.401409 22186 raft_consensus.cc:2229] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:16.401893 22186 raft_consensus.cc:2258] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:16.401944 22188 tablet_replica.cc:330] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: stopping tablet replica
I20260808 02:24:16.402338 22188 raft_consensus.cc:2229] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:16.402863 22188 raft_consensus.cc:2258] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:16.403131 22186 ts_tablet_manager.cc:1905] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:16.403890 22187 tablet_replica.cc:330] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: stopping tablet replica
I20260808 02:24:16.404203 22187 raft_consensus.cc:2229] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:16.404410 22188 ts_tablet_manager.cc:1905] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:16.405056 22187 raft_consensus.cc:2258] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:16.406276 22187 ts_tablet_manager.cc:1905] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:16.412047 22186 ts_tablet_manager.cc:1918] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:16.412227 22186 log.cc:1196] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/efb2e7047913495fa08be86806786d5d
I20260808 02:24:16.412791 22186 ts_tablet_manager.cc:1939] T efb2e7047913495fa08be86806786d5d P 0eba9a494fc24e43be6a76f3ebc7d5ba: Deleting consensus metadata
I20260808 02:24:16.417089 22187 ts_tablet_manager.cc:1918] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:16.417443 22188 ts_tablet_manager.cc:1918] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:16.417678 22188 log.cc:1196] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/efb2e7047913495fa08be86806786d5d
I20260808 02:24:16.418447 22188 ts_tablet_manager.cc:1939] T efb2e7047913495fa08be86806786d5d P 8ae1994a90704413b2310401dabc974d: Deleting consensus metadata
I20260808 02:24:16.420010 22187 log.cc:1196] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/efb2e7047913495fa08be86806786d5d
I20260808 02:24:16.420185 21568 catalog_manager.cc:4841] TS 8ae1994a90704413b2310401dabc974d (127.19.231.3:46875): tablet efb2e7047913495fa08be86806786d5d (table test-table [id=2f629cfef7ac4b22974e861db3aaeeed]) successfully deleted
I20260808 02:24:16.421139 21567 catalog_manager.cc:4841] TS 0eba9a494fc24e43be6a76f3ebc7d5ba (127.19.231.1:41735): tablet efb2e7047913495fa08be86806786d5d (table test-table [id=2f629cfef7ac4b22974e861db3aaeeed]) successfully deleted
I20260808 02:24:16.421509 22187 ts_tablet_manager.cc:1939] T efb2e7047913495fa08be86806786d5d P ded9a9a4750947b5b65fd3be9341cf8b: Deleting consensus metadata
I20260808 02:24:16.422716 21568 catalog_manager.cc:4841] TS ded9a9a4750947b5b65fd3be9341cf8b (127.19.231.2:42655): tablet efb2e7047913495fa08be86806786d5d (table test-table [id=2f629cfef7ac4b22974e861db3aaeeed]) successfully deleted
Aug 08 02:24:16 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:16.439371 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.412960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.19.231.1:41735)
0808 02:24:16.421113 (+ 8153us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.421120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.421911 (+ 791us) server_negotiation.cc:407] Connection header received
0808 02:24:16.422165 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.422172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.422291 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.422418 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:16.424705 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.425620 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.427225 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.427782 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.427933 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.432656 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:16.432672 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:16.432689 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:16.432723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:16.436082 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.436842 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.436850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.436853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.437014 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.437419 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.437423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.437425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.437776 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:16.438134 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.438463 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.438825 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7981,"thread_start_us":7846,"threads_started":1}
W20260808 02:24:16.440193 21798 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36924
I20260808 02:24:16.464267 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.443388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.19.231.62:34655)
0808 02:24:16.444612 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.444619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.444643 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:24:16.444714 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.444721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.444868 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.445000 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:16.446783 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.448568 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.450108 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.450891 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.451039 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.454687 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:16.454695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:16.454705 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:16.454761 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:16.460588 (+ 5827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.461299 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.461305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.461310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.461639 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.462047 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.462052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.462056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.462688 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:16.463005 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.463333 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.463636 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":755,"threads_started":1}
W20260808 02:24:16.465157 21586 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59128
I20260808 02:24:16.468138 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21658
I20260808 02:24:16.488035 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.469402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.19.231.1:41735)
0808 02:24:16.469667 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.469673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.471862 (+ 2189us) server_negotiation.cc:407] Connection header received
0808 02:24:16.472040 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.472046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.472156 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.472269 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:16.473993 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.474855 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.476471 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.476952 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.477109 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.480336 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:16.480343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:16.480354 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:16.480392 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:16.484482 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.485259 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.485266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.485270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.485463 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.485956 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.485962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.485966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.486345 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:16.486788 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.487141 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.487489 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260808 02:24:16.488946 21798 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36926
W20260808 02:24:16.489743 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:16.490072 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 21658
I20260808 02:24:16.513522 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21807
Aug 08 02:24:16 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:16.533257 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.514346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.19.231.2:42655)
0808 02:24:16.514564 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.514568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.514604 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:24:16.514817 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.514821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.514884 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.514976 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:16.517159 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.517990 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.520171 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.520699 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.520796 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.526573 (+ 5777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:16.526585 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:16.526600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:16.526697 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:16.529969 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.530900 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.530911 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.530913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.531063 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.531491 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.531495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.531497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.531903 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:16.532253 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.532639 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.532900 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 02:24:16.534142 21943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50752
W20260808 02:24:16.535086 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:16.535276 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 21807
I20260808 02:24:16.560895 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21952
Aug 08 02:24:16 dist-test-slave-pgkr krb5kdc[21535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155854, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:16.576944 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:16.561809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.19.231.3:46875)
0808 02:24:16.562595 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:16.562600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:16.562683 (+ 83us) server_negotiation.cc:407] Connection header received
0808 02:24:16.562751 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:16.562755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:16.562817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:16.562935 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:16.564176 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.565115 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.566414 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:16.567076 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:16.567180 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:16.571383 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:16.571398 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:16.571414 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:16.571443 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:16.574129 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.574857 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.574870 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.574873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.575027 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:16.575395 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:16.575400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:16.575403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:16.575762 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:16.576076 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:16.576370 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 02:24:16.576596 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":422,"threads_started":1}
W20260808 02:24:16.577706 22092 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48326
W20260808 02:24:16.578294 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21952: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:16.578516 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 21952
I20260808 02:24:16.604997 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 21548
I20260808 02:24:16.606562 21626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:16.725683 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 21548
2026-08-08T02:24:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6736 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:16 dist-test-slave-pgkr krb5kdc[22200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:16 dist-test-slave-pgkr krb5kdc[22200](info): set up 2 sockets
Aug 08 02:24:16 dist-test-slave-pgkr krb5kdc[22200](info): commencing operation
krb5kdc: starting...
W20260808 02:24:17.866953 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:18 dist-test-slave-pgkr krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:18.079816 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37327
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:18.457800 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:18.458194 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:18.458377 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:18.467160 22216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:18.467355 22216 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:24:18.467473 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:18.467628 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:18.467747 22216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:18.467886 22216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:18.480516 22216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37327
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:18.482002 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:18.483946 22216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:18.498893 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:18.498937 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:18.498818 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:18.500524 22216 server_base.cc:1034] running on GCE node
I20260808 02:24:18.501443 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:18.503296 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:18.504622 22216 hybrid_clock.cc:648] HybridClock initialized: now 1786155858504509 us; error 52 us; skew 500 ppm
I20260808 02:24:18.505184 22216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:18 dist-test-slave-pgkr krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:18.517014 22216 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:18.519104 22216 webserver.cc:460] Webserver started at http://127.19.231.62:41077/ using document root <none> and password file <none>
I20260808 02:24:18.519932 22216 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:18.520037 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:18.520367 22216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:18.524305 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "95b0c5ea53e54cea878e0c94db41866a"
format_stamp: "Formatted at 2026-08-08 02:24:18 on dist-test-slave-pgkr"
server_key: "0a680148c0e47f5c2a123bcc3afe3c7e"
server_key_iv: "6c3a6c9e756bdd8832e360099a826e8c"
server_key_version: "encryptionkey@0"
I20260808 02:24:18.525255 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "95b0c5ea53e54cea878e0c94db41866a"
format_stamp: "Formatted at 2026-08-08 02:24:18 on dist-test-slave-pgkr"
server_key: "0a680148c0e47f5c2a123bcc3afe3c7e"
server_key_iv: "6c3a6c9e756bdd8832e360099a826e8c"
server_key_version: "encryptionkey@0"
I20260808 02:24:18.531569 22216 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 02:24:18.535794 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:18.537240 22216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:24:18.537451 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "95b0c5ea53e54cea878e0c94db41866a"
format_stamp: "Formatted at 2026-08-08 02:24:18 on dist-test-slave-pgkr"
server_key: "0a680148c0e47f5c2a123bcc3afe3c7e"
server_key_iv: "6c3a6c9e756bdd8832e360099a826e8c"
server_key_version: "encryptionkey@0"
I20260808 02:24:18.537690 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:18.560608 22216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:18.564824 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:18.565133 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:18.589676 22216 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:37327
I20260808 02:24:18.589700 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:37327 every 8 connection(s)
I20260808 02:24:18.591251 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:18.592698 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 22216
I20260808 02:24:18.592928 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:18.593789 20380 external_mini_cluster.cc:1428] Setting key 20422b62eace5576003811e610d41654
I20260808 02:24:18.595980 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:24:18 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:18.610996 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Bootstrap starting.
I20260808 02:24:18.616354 22296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:18.617982 22296 log.cc:824] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:18.620985 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:18.595898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.19.231.62:37327)
0808 02:24:18.596767 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:18.596780 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:18.596821 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:24:18.598488 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:18.598511 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:18.599000 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:18.599359 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:18.601226 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:18.602736 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:18.604510 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:18.605193 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:18.605332 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:18.610875 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:18.610887 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:18.610906 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:18.610953 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:18.615677 (+ 4724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:18.616400 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:18.616411 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:18.616421 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:18.616684 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:18.617089 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:18.617095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:18.617099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:18.617942 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:18.618320 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:18.618874 (+ 554us) server_negotiation.cc:299] Negotiation successful
0808 02:24:18.619408 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":460,"threads_started":1}
I20260808 02:24:18.621940 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: No bootstrap required, opened a new log
I20260808 02:24:18.631398 22296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b0c5ea53e54cea878e0c94db41866a" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37327 } }
I20260808 02:24:18.631940 22296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:18.632073 22296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b0c5ea53e54cea878e0c94db41866a, State: Initialized, Role: FOLLOWER
I20260808 02:24:18.632866 22296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b0c5ea53e54cea878e0c94db41866a" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37327 } }
I20260808 02:24:18.633118 22296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:18.633302 22296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:18.633540 22296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:18.636386 22296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b0c5ea53e54cea878e0c94db41866a" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37327 } }
I20260808 02:24:18.637243 22296 leader_election.cc:302] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b0c5ea53e54cea878e0c94db41866a; no voters:
I20260808 02:24:18.637930 22296 leader_election.cc:288] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:18.638080 22301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:18.639393 22301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [term 1 LEADER]: Becoming Leader. State: Replica: 95b0c5ea53e54cea878e0c94db41866a, State: Running, Role: LEADER
I20260808 02:24:18.640051 22301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b0c5ea53e54cea878e0c94db41866a" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37327 } }
I20260808 02:24:18.642243 22296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:18.643442 22302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b0c5ea53e54cea878e0c94db41866a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b0c5ea53e54cea878e0c94db41866a" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37327 } } }
I20260808 02:24:18.643953 22302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:18.644748 22303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b0c5ea53e54cea878e0c94db41866a. Latest consensus state: current_term: 1 leader_uuid: "95b0c5ea53e54cea878e0c94db41866a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b0c5ea53e54cea878e0c94db41866a" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37327 } } }
I20260808 02:24:18.645148 22303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:18.646996 22306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:18.654376 22306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:18.664600 22306 catalog_manager.cc:1347] Generated new cluster ID: 37c0fbedc7bb4ae3aaac213577329e68
I20260808 02:24:18.664734 22306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:18.691440 22306 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:18.693326 22306 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:18.718945 22306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Generated new TSK 0
I20260808 02:24:18.719874 22306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:18.786700 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37327
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:19.127439 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:19.127758 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:19.127933 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:19.135244 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:19.135409 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:19.135553 22322 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:24:19.135645 22322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:24:19.135780 22322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:19.146008 22322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:37327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:19.147459 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:19.148816 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:19.163565 22329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:19.163956 22328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:19.163604 22331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:19.164870 22322 server_base.cc:1034] running on GCE node
I20260808 02:24:19.165598 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:19.166656 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:19.167924 22322 hybrid_clock.cc:648] HybridClock initialized: now 1786155859167864 us; error 49 us; skew 500 ppm
I20260808 02:24:19.168264 22322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:19 dist-test-slave-pgkr krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155859, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:19.171550 22322 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:19.173105 22322 webserver.cc:460] Webserver started at http://127.19.231.1:42117/ using document root <none> and password file <none>
I20260808 02:24:19.173794 22322 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:19.173943 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:19.174335 22322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:19.177716 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "7b347aa5f1b8470b9d38d04cc4373637"
format_stamp: "Formatted at 2026-08-08 02:24:19 on dist-test-slave-pgkr"
server_key: "e2fbd5358fca52927c5dfcd627a1f705"
server_key_iv: "aff1b2e632afe4ac8b0a001d3ad179d7"
server_key_version: "encryptionkey@0"
I20260808 02:24:19.178499 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "7b347aa5f1b8470b9d38d04cc4373637"
format_stamp: "Formatted at 2026-08-08 02:24:19 on dist-test-slave-pgkr"
server_key: "e2fbd5358fca52927c5dfcd627a1f705"
server_key_iv: "aff1b2e632afe4ac8b0a001d3ad179d7"
server_key_version: "encryptionkey@0"
I20260808 02:24:19.183928 22322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:24:19.187562 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:19.189069 22322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:19.189266 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "7b347aa5f1b8470b9d38d04cc4373637"
format_stamp: "Formatted at 2026-08-08 02:24:19 on dist-test-slave-pgkr"
server_key: "e2fbd5358fca52927c5dfcd627a1f705"
server_key_iv: "aff1b2e632afe4ac8b0a001d3ad179d7"
server_key_version: "encryptionkey@0"
I20260808 02:24:19.189468 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:19.207319 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:19.212086 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:19.212407 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:19.215863 22322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:19.216018 22322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:19.216177 22322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:19.216395 22322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 02:24:19 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155859, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:19.250425 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.216757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37327 (local address 127.19.231.1:54443)
0808 02:24:19.217677 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:19.217741 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:19.219722 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:19.220526 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:19.220534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:19.221141 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:19.221761 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.221779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.223262 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.223265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.225469 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.225482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.226532 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.226540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.226828 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.227767 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:19.227810 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:19.238508 (+ 10698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:19.243431 (+ 4923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.243445 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.243466 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.244369 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.244903 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.244915 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.244921 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.245294 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.246300 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:19.246316 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:19.247004 (+ 688us) client_negotiation.cc:769] Sending connection context
0808 02:24:19.247468 (+ 464us) client_negotiation.cc:240] Negotiation successful
0808 02:24:19.248267 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":426,"threads_started":1}
I20260808 02:24:19.251956 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.216058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:54443 (local address 127.19.231.62:37327)
0808 02:24:19.216613 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:19.216621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:19.217903 (+ 1282us) server_negotiation.cc:407] Connection header received
0808 02:24:19.220025 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:19.220044 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:19.220157 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:19.220316 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:19.221936 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.223080 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.225625 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.226253 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.226389 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.238908 (+ 12519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:19.238921 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:19.238930 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:19.238999 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:19.243223 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.244501 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.244510 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.244516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.244712 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.245488 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.245495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.245501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.245971 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:19.246198 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:19.251482 (+ 5284us) server_negotiation.cc:299] Negotiation successful
0808 02:24:19.251691 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":282,"threads_started":1}
I20260808 02:24:19.282881 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.263616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:45557 (local address 127.19.231.62:37327)
0808 02:24:19.263807 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:19.263811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:19.265193 (+ 1382us) server_negotiation.cc:407] Connection header received
0808 02:24:19.265585 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:19.265593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:19.265681 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:19.265817 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:19.267768 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.268912 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.271831 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.272505 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.272646 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.274891 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:19.274900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:19.274906 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:19.274958 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:19.279252 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.280394 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.280401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.280408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.280557 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.280999 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.281005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.281015 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.281480 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:19.281685 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:19.282152 (+ 467us) server_negotiation.cc:299] Negotiation successful
0808 02:24:19.282424 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:24:19.283068 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.263939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37327 (local address 127.19.231.1:45557)
0808 02:24:19.265054 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:19.265083 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:19.265361 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:19.265848 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:19.265855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:19.266824 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:19.267567 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.267593 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.269163 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.269183 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.271453 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.271481 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.272727 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.272734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.272907 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.273786 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:19.273804 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:19.274682 (+ 878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:19.279473 (+ 4791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.279482 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.279489 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.280173 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.280697 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.280703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.280707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.280859 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.281636 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:19.281641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:19.281882 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 02:24:19.282042 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 02:24:19.282451 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":612,"threads_started":1}
I20260808 02:24:19.284588 22322 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:46557
I20260808 02:24:19.284631 22464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:46557 every 8 connection(s)
I20260808 02:24:19.286152 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:19.289041 22465 heartbeater.cc:347] Connected to a master server at 127.19.231.62:37327
I20260808 02:24:19.289305 22465 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:19.290043 22465 heartbeater.cc:511] Master 127.19.231.62:37327 requested a full tablet report, sending...
I20260808 02:24:19.292321 22250 ts_manager.cc:194] Registered new tserver with Master: 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557)
I20260808 02:24:19.292455 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 22322
I20260808 02:24:19.292702 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:19.293388 20380 external_mini_cluster.cc:1428] Setting key c8d1ff1fa5e078b85677d6fc0d8bdd2f
I20260808 02:24:19.294667 22250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:54443
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:19.342096 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37327
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:19.675771 22469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:19.676093 22469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:19.676276 22469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:19.683826 22469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:19.683972 22469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:19.684141 22469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:24:19.684222 22469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:24:19.684336 22469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:19.694819 22469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:37327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:19.696133 22469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:19.697481 22469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:19.711028 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:19.711114 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:19.711189 22475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:19.712306 22469 server_base.cc:1034] running on GCE node
I20260808 02:24:19.712953 22469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:19.713980 22469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:19.715237 22469 hybrid_clock.cc:648] HybridClock initialized: now 1786155859715172 us; error 65 us; skew 500 ppm
I20260808 02:24:19.715571 22469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:19 dist-test-slave-pgkr krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155859, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:19.718784 22469 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:19.720376 22469 webserver.cc:460] Webserver started at http://127.19.231.2:34843/ using document root <none> and password file <none>
I20260808 02:24:19.721091 22469 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:19.721238 22469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:19.721567 22469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:19.724584 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "5a345ed5dae04e3fb0428e1387c81a73"
format_stamp: "Formatted at 2026-08-08 02:24:19 on dist-test-slave-pgkr"
server_key: "780f1376c0c0d58514759bda93f62935"
server_key_iv: "67d759931cff74f2f0a2b9b5dd737f23"
server_key_version: "encryptionkey@0"
I20260808 02:24:19.725345 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "5a345ed5dae04e3fb0428e1387c81a73"
format_stamp: "Formatted at 2026-08-08 02:24:19 on dist-test-slave-pgkr"
server_key: "780f1376c0c0d58514759bda93f62935"
server_key_iv: "67d759931cff74f2f0a2b9b5dd737f23"
server_key_version: "encryptionkey@0"
I20260808 02:24:19.731065 22469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:24:19.734663 22485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:19.736065 22469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:19.736248 22469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "5a345ed5dae04e3fb0428e1387c81a73"
format_stamp: "Formatted at 2026-08-08 02:24:19 on dist-test-slave-pgkr"
server_key: "780f1376c0c0d58514759bda93f62935"
server_key_iv: "67d759931cff74f2f0a2b9b5dd737f23"
server_key_version: "encryptionkey@0"
I20260808 02:24:19.736460 22469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:19.751307 22469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:19.755767 22469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:19.756062 22469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:19.758946 22469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:19.759068 22469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:19.759217 22469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:19.759331 22469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:24:19 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155859, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:19.785281 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.760133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37327 (local address 127.19.231.2:55795)
0808 02:24:19.760964 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:19.761003 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:19.762934 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:19.763713 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:19.763720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:19.764470 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:19.765311 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.765333 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.767093 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.767107 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.769255 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.769267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.770100 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.770108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.770334 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.771141 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:19.771176 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:19.774510 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:19.778860 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.778873 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.778895 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.779654 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.780215 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.780223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.780230 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.780600 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.781569 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:19.781583 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:19.782157 (+ 574us) client_negotiation.cc:769] Sending connection context
0808 02:24:19.782538 (+ 381us) client_negotiation.cc:240] Negotiation successful
0808 02:24:19.783238 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":345,"threads_started":1}
I20260808 02:24:19.785750 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.759411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:55795 (local address 127.19.231.62:37327)
0808 02:24:19.759618 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:19.759626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:19.761123 (+ 1497us) server_negotiation.cc:407] Connection header received
0808 02:24:19.763252 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:19.763257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:19.763357 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:19.763469 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:19.765607 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.766914 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.769408 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.769855 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.769992 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.774767 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:19.774773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:19.774776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:19.774847 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:19.778701 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.779857 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.779864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.779871 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.780038 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.780792 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.780800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.780806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.781278 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:19.781487 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:19.785141 (+ 3654us) server_negotiation.cc:299] Negotiation successful
0808 02:24:19.785391 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:24:19.809181 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.791915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:52401 (local address 127.19.231.62:37327)
0808 02:24:19.792139 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:19.792157 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:19.792787 (+ 630us) server_negotiation.cc:407] Connection header received
0808 02:24:19.793018 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:19.793025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:19.793131 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:19.793245 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:19.795138 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.796460 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.798653 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.799105 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.799270 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.802241 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:19.802249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:19.802255 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:19.802291 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:19.805679 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.806683 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.806691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.806696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.806878 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:19.807396 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:19.807404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:19.807410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:19.807942 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:19.808114 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:19.808755 (+ 641us) server_negotiation.cc:299] Negotiation successful
0808 02:24:19.808930 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:24:19.809813 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:19.791898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37327 (local address 127.19.231.2:52401)
0808 02:24:19.792669 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:19.792688 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:19.792863 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:19.793465 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:19.793469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:19.794286 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:19.794957 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.794976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.796606 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.796613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.798471 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:19.798484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:19.799343 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:19.799353 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:19.799644 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:19.800355 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:19.800388 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:19.801995 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:19.805813 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.805819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.805826 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.806454 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.807009 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:19.807014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:19.807021 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:19.807251 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:19.808124 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:19.808132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:19.808388 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 02:24:19.808608 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 02:24:19.809106 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":391,"threads_started":1}
I20260808 02:24:19.813658 22469 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:39549
I20260808 02:24:19.813710 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:39549 every 8 connection(s)
I20260808 02:24:19.814994 22469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:19.817051 22611 heartbeater.cc:347] Connected to a master server at 127.19.231.62:37327
I20260808 02:24:19.817257 22611 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:19.817838 22611 heartbeater.cc:511] Master 127.19.231.62:37327 requested a full tablet report, sending...
I20260808 02:24:19.819072 22250 ts_manager.cc:194] Registered new tserver with Master: 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:19.819921 22250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:55795
I20260808 02:24:19.823634 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 22469
I20260808 02:24:19.823819 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:19.824288 20380 external_mini_cluster.cc:1428] Setting key 5225395ceaeaffaf3e5fb1f0b9dc031f
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:19.872323 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37327
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:20.206378 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:20.206734 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:20.206923 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:20.214177 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:20.214331 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:20.214486 22615 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 02:24:20.214586 22615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:24:20.214740 22615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:20.224367 22615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:37327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:20.225716 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:20.227108 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:20.240744 22622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:20.240983 22624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:20.241319 22621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:20.241923 22615 server_base.cc:1034] running on GCE node
I20260808 02:24:20.242655 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:20.243696 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:20.244904 22615 hybrid_clock.cc:648] HybridClock initialized: now 1786155860244862 us; error 37 us; skew 500 ppm
I20260808 02:24:20.245221 22615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:20 dist-test-slave-pgkr krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:20.248646 22615 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:20.250177 22615 webserver.cc:460] Webserver started at http://127.19.231.3:40427/ using document root <none> and password file <none>
I20260808 02:24:20.251101 22615 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:20.251257 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:20.251643 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:20.255069 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "d356e1b9aa504442b4837001d5fdf804"
format_stamp: "Formatted at 2026-08-08 02:24:20 on dist-test-slave-pgkr"
server_key: "b239a1d48b6e8106f353835d3ebb933d"
server_key_iv: "a77235712250abeb9741693850634f86"
server_key_version: "encryptionkey@0"
I20260808 02:24:20.255923 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "d356e1b9aa504442b4837001d5fdf804"
format_stamp: "Formatted at 2026-08-08 02:24:20 on dist-test-slave-pgkr"
server_key: "b239a1d48b6e8106f353835d3ebb933d"
server_key_iv: "a77235712250abeb9741693850634f86"
server_key_version: "encryptionkey@0"
I20260808 02:24:20.261436 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:20.265256 22632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:20.266703 22615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:20.266888 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "d356e1b9aa504442b4837001d5fdf804"
format_stamp: "Formatted at 2026-08-08 02:24:20 on dist-test-slave-pgkr"
server_key: "b239a1d48b6e8106f353835d3ebb933d"
server_key_iv: "a77235712250abeb9741693850634f86"
server_key_version: "encryptionkey@0"
I20260808 02:24:20.267107 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:20.294850 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:20.298311 22465 heartbeater.cc:503] Master 127.19.231.62:37327 was elected leader, sending a full tablet report...
I20260808 02:24:20.299456 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:20.299801 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:20.302598 22615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:20.302799 22615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:20.302933 22615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:20.303041 22615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:24:20 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:20.327123 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.303278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:44925 (local address 127.19.231.62:37327)
0808 02:24:20.303553 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.303564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.304898 (+ 1334us) server_negotiation.cc:407] Connection header received
0808 02:24:20.307019 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.307029 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.307164 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.307287 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:20.309197 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.310595 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.314356 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.314997 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.315149 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.320429 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:20.320436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:20.320441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:20.320484 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:20.323767 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.324760 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.324769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.324774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.324906 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.325342 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.325348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.325351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.325618 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:20.325777 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.326495 (+ 718us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.326748 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 02:24:20.328720 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.303821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37327 (local address 127.19.231.3:44925)
0808 02:24:20.304710 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.304768 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.306733 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.307525 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.307533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.308049 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:20.308894 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.308917 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.310917 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.310924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.314191 (+ 3267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.314209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.315307 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.315319 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.315705 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.316688 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:20.316712 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:20.320193 (+ 3481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:20.323910 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:20.323919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:20.323934 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:20.324628 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:20.325026 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:20.325031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:20.325035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:20.325230 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:20.325756 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:20.325763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:20.326111 (+ 348us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.326372 (+ 261us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.326984 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":382,"threads_started":1}
I20260808 02:24:20.353363 22615 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:46031
I20260808 02:24:20.353825 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.336197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37327 (local address 127.19.231.3:39907)
0808 02:24:20.336895 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.336916 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.337051 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.337455 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.337459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.337887 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:20.338166 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.338175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.339880 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.339895 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.342097 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.342120 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.343300 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.343310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.343443 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.344320 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:20.344345 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:20.345143 (+ 798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:20.349804 (+ 4661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:20.349812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:20.349818 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:20.350548 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:20.351174 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:20.351181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:20.351186 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:20.351388 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:20.352000 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:20.352008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:20.352280 (+ 272us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.352473 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.353019 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":526,"threads_started":1}
I20260808 02:24:20.353479 22757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:46031 every 8 connection(s)
I20260808 02:24:20.354491 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.336657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:39907 (local address 127.19.231.62:37327)
0808 02:24:20.336911 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.336921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.336991 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:24:20.337185 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.337195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.337304 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.337451 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:20.338313 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.339641 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.342282 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.343033 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.343174 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.345362 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:20.345373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:20.345378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:20.345416 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:20.349640 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.350786 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.350794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.350800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.350996 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.351534 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.351537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.351539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.351800 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:20.353604 (+ 1804us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.353758 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.354016 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:24:20.355166 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:20.357530 22758 heartbeater.cc:347] Connected to a master server at 127.19.231.62:37327
I20260808 02:24:20.357941 22758 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:20.359047 22758 heartbeater.cc:511] Master 127.19.231.62:37327 requested a full tablet report, sending...
I20260808 02:24:20.360363 22250 ts_manager.cc:194] Registered new tserver with Master: d356e1b9aa504442b4837001d5fdf804 (127.19.231.3:46031)
I20260808 02:24:20.361176 22250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:44925
I20260808 02:24:20.364617 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 22615
I20260808 02:24:20.364761 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:20.365177 20380 external_mini_cluster.cc:1428] Setting key 98138bfea144ab2cd979a9771491b917
I20260808 02:24:20.368027 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:24:20 dist-test-slave-pgkr krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:24:20 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:20.427090 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.407231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.19.231.62:37327)
0808 02:24:20.407477 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.407481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.407563 (+ 82us) server_negotiation.cc:407] Connection header received
0808 02:24:20.407845 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.407850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.407987 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.408107 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:20.409920 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.411155 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.412814 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.413420 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.413529 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.419454 (+ 5925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:20.419460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:20.419463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:20.419501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:20.423288 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.424182 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.424193 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.424196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.424394 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.424930 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.424939 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.424943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.425344 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:20.425778 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.426069 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.426386 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:24:20.432894 22250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:20.441897 22250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:20.485950 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.458735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46557 (local address 127.0.0.1:43204)
0808 02:24:20.472948 (+ 14213us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.472976 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.473157 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.474972 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.474977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.475001 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:20.476297 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.476310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.478926 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.478934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.482364 (+ 3430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.482381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.484806 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.484812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.485021 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.485104 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.485262 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.485501 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13981,"mutex_wait_us":26,"thread_start_us":15056,"threads_started":1}
I20260808 02:24:20.485925 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.459213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.19.231.1:46557)
0808 02:24:20.474065 (+ 14852us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.474081 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.474122 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:24:20.474281 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.474289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.474718 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.474995 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:20.476698 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.478797 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.482702 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.484585 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.484770 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.485072 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.485311 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.485511 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14618,"thread_start_us":14283,"threads_started":1}
I20260808 02:24:20.488677 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.458689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.19.231.3:46031)
0808 02:24:20.472935 (+ 14246us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.472946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.473026 (+ 80us) server_negotiation.cc:407] Connection header received
0808 02:24:20.473198 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.473214 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.473643 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.473929 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:20.476910 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.479939 (+ 3029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.484162 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.486390 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.486545 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.486940 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.487789 (+ 849us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.487987 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13910,"thread_start_us":13522,"threads_started":1}
I20260808 02:24:20.488849 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.458049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:46031 (local address 127.0.0.1:60742)
0808 02:24:20.465267 (+ 7218us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.465325 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.465600 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.474765 (+ 9165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.474772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.474819 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:20.476267 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.476284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.480157 (+ 3873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.480163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.483987 (+ 3824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.484003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.487266 (+ 3263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.487277 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.487553 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.487578 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.487688 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.487845 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1944,"mutex_wait_us":29,"thread_start_us":744,"threads_started":1}
I20260808 02:24:20.489446 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.460208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.19.231.2:39549)
0808 02:24:20.468829 (+ 8621us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.468843 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.474908 (+ 6065us) server_negotiation.cc:407] Connection header received
0808 02:24:20.475012 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.475018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.475366 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.475616 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:20.478018 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.480110 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.484073 (+ 3963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.486756 (+ 2683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.486924 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.488443 (+ 1519us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.488575 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.488736 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8407,"thread_start_us":8122,"threads_started":1}
I20260808 02:24:20.489648 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.459616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:39549 (local address 127.0.0.1:49228)
0808 02:24:20.474571 (+ 14955us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.474586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.474781 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.475629 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.475634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.475676 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:20.477032 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.477046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.480292 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.480298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.483882 (+ 3584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.483894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.487265 (+ 3371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.487273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.487451 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.487483 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.487565 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.487697 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14789,"mutex_wait_us":443,"thread_start_us":13227,"threads_started":1}
I20260808 02:24:20.491303 22391 tablet_service.cc:1467] Processing CreateTablet for tablet 6474499fdb4a40a1873365f4d5328da7 (DEFAULT_TABLE table=test-table [id=9e0579eef0ab479983ba0335b912bfc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:20.492760 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6474499fdb4a40a1873365f4d5328da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:20.492921 22537 tablet_service.cc:1467] Processing CreateTablet for tablet 6474499fdb4a40a1873365f4d5328da7 (DEFAULT_TABLE table=test-table [id=9e0579eef0ab479983ba0335b912bfc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:20.494287 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6474499fdb4a40a1873365f4d5328da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:20.494092 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 6474499fdb4a40a1873365f4d5328da7 (DEFAULT_TABLE table=test-table [id=9e0579eef0ab479983ba0335b912bfc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:20.495536 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6474499fdb4a40a1873365f4d5328da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:20.504588 22777 tablet_bootstrap.cc:492] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Bootstrap starting.
I20260808 02:24:20.508865 22777 tablet_bootstrap.cc:654] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:20.509941 22777 log.cc:824] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:20.512522 22777 tablet_bootstrap.cc:492] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: No bootstrap required, opened a new log
I20260808 02:24:20.512810 22777 ts_tablet_manager.cc:1397] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 02:24:20.515400 22780 tablet_bootstrap.cc:492] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Bootstrap starting.
I20260808 02:24:20.516134 22778 tablet_bootstrap.cc:492] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Bootstrap starting.
I20260808 02:24:20.519477 22778 tablet_bootstrap.cc:654] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:20.519533 22780 tablet_bootstrap.cc:654] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:20.520565 22778 log.cc:824] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:20.520733 22780 log.cc:824] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:20.523247 22777 raft_consensus.cc:348] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.523703 22777 raft_consensus.cc:374] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:20.523816 22777 raft_consensus.cc:729] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b347aa5f1b8470b9d38d04cc4373637, State: Initialized, Role: FOLLOWER
I20260808 02:24:20.524497 22777 consensus_queue.cc:260] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.524863 22780 tablet_bootstrap.cc:492] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: No bootstrap required, opened a new log
I20260808 02:24:20.524869 22778 tablet_bootstrap.cc:492] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: No bootstrap required, opened a new log
I20260808 02:24:20.525185 22778 ts_tablet_manager.cc:1397] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260808 02:24:20.525184 22780 ts_tablet_manager.cc:1397] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 02:24:20.526372 22777 ts_tablet_manager.cc:1428] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260808 02:24:20.534153 22780 raft_consensus.cc:348] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.534534 22780 raft_consensus.cc:374] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:20.534691 22780 raft_consensus.cc:729] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a345ed5dae04e3fb0428e1387c81a73, State: Initialized, Role: FOLLOWER
I20260808 02:24:20.535319 22780 consensus_queue.cc:260] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.535409 22778 raft_consensus.cc:348] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.535831 22778 raft_consensus.cc:374] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:20.535974 22778 raft_consensus.cc:729] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Initialized, Role: FOLLOWER
I20260808 02:24:20.536401 22611 heartbeater.cc:503] Master 127.19.231.62:37327 was elected leader, sending a full tablet report...
I20260808 02:24:20.536633 22778 consensus_queue.cc:260] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.536891 22780 ts_tablet_manager.cc:1428] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260808 02:24:20.537683 22758 heartbeater.cc:503] Master 127.19.231.62:37327 was elected leader, sending a full tablet report...
I20260808 02:24:20.538401 22778 ts_tablet_manager.cc:1428] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260808 02:24:20.539168 22466 tablet.cc:2369] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:24:20.566985 22612 tablet.cc:2369] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:24:20.608733 22759 tablet.cc:2369] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:20.639644 22786 raft_consensus.cc:482] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:20.640089 22786 raft_consensus.cc:504] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.642231 22786 leader_election.cc:288] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549), 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557)
I20260808 02:24:20.649049 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.642772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:39549 (local address 127.19.231.3:37985)
0808 02:24:20.643071 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.643094 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.643238 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.643579 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.643584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.643611 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:20.643836 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.643845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.644960 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.644972 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.646408 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.646425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.648242 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.648250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.648461 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.648497 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.648598 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.648752 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260808 02:24:20.649509 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.642846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:37985 (local address 127.19.231.2:39549)
0808 02:24:20.643137 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.643144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.643178 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:24:20.643399 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.643403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.643463 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.643559 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:20.643978 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.644815 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.646668 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.647876 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.648052 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.648146 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.648663 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.648863 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260808 02:24:20.650085 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.642835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:46257 (local address 127.19.231.1:46557)
0808 02:24:20.643086 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.643093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.643805 (+ 712us) server_negotiation.cc:407] Connection header received
0808 02:24:20.644019 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.644030 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.644138 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.644233 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:20.644740 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.645423 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.647788 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.649047 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.649219 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.649291 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.649796 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.649907 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 02:24:20.650210 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.642821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46557 (local address 127.19.231.3:46257)
0808 02:24:20.643679 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.643700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.643870 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.644265 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.644270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.644295 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:20.644593 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.644601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.645557 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.645564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.647565 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.647582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.649357 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.649363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.649547 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.649584 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.649753 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.649877 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":487,"threads_started":1}
I20260808 02:24:20.650492 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6474499fdb4a40a1873365f4d5328da7" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73" is_pre_election: true
I20260808 02:24:20.651119 22565 raft_consensus.cc:2454] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:20.651321 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6474499fdb4a40a1873365f4d5328da7" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637" is_pre_election: true
I20260808 02:24:20.651777 22636 leader_election.cc:302] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a345ed5dae04e3fb0428e1387c81a73, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:20.651811 22418 raft_consensus.cc:2454] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:20.652210 22786 raft_consensus.cc:2781] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:20.652395 22786 raft_consensus.cc:482] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:20.652532 22786 raft_consensus.cc:3037] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:20.654983 22786 raft_consensus.cc:504] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.655664 22786 leader_election.cc:288] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Requested vote from peers 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549), 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557)
I20260808 02:24:20.655987 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6474499fdb4a40a1873365f4d5328da7" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73"
I20260808 02:24:20.656103 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6474499fdb4a40a1873365f4d5328da7" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637"
I20260808 02:24:20.656195 22565 raft_consensus.cc:3037] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:20.656333 22418 raft_consensus.cc:3037] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:20.658382 22565 raft_consensus.cc:2454] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:20.658533 22418 raft_consensus.cc:2454] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:20.658771 22636 leader_election.cc:302] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a345ed5dae04e3fb0428e1387c81a73, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:20.659072 22786 raft_consensus.cc:2781] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:20.659525 22786 raft_consensus.cc:686] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Becoming Leader. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Running, Role: LEADER
I20260808 02:24:20.660027 22786 consensus_queue.cc:237] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } }
I20260808 02:24:20.664860 22249 catalog_manager.cc:5495] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 reported cstate change: term changed from 0 to 1, leader changed from <none> to d356e1b9aa504442b4837001d5fdf804 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:24:20 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:20.722796 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.705246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.19.231.62:37327)
0808 02:24:20.705420 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.705424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.706145 (+ 721us) server_negotiation.cc:407] Connection header received
0808 02:24:20.706433 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.706439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.706550 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.706710 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:20.708304 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.709694 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.711769 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.712561 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.712722 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.717693 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:20.717698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:20.717701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:20.717729 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:20.720178 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.720913 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.720917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.720919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.721032 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:20.721350 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:20.721353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:20.721355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:20.721707 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:20.721853 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.722248 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.722477 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":62}
I20260808 02:24:20.722815 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.705323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37327 (local address 127.0.0.1:46466)
0808 02:24:20.706037 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.706059 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.706289 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.706741 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.706746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.707736 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:20.708112 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.708124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.709846 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.709852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.711603 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.711617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.712851 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.712861 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.713093 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.714323 (+ 1230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:20.714358 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:20.717489 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:20.720309 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:20.720320 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:20.720326 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:20.720813 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:20.721121 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:20.721125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:20.721127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:20.721258 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:20.721829 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:20.721835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:20.722009 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.722158 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.722363 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":345,"threads_started":1}
I20260808 02:24:20.728232 22249 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:46466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:24:20.729305 22249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:20.735973 22391 tablet_service.cc:1467] Processing CreateTablet for tablet 3d642ee20bab4e259c492bb3346a89fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1aaeb5f9207b4b1fa2a1c3ed0d6ea819]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:24:20.736124 22537 tablet_service.cc:1467] Processing CreateTablet for tablet 3d642ee20bab4e259c492bb3346a89fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1aaeb5f9207b4b1fa2a1c3ed0d6ea819]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:24:20.736277 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 3d642ee20bab4e259c492bb3346a89fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1aaeb5f9207b4b1fa2a1c3ed0d6ea819]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:24:20.736912 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d642ee20bab4e259c492bb3346a89fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:20.737036 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d642ee20bab4e259c492bb3346a89fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:20.737036 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d642ee20bab4e259c492bb3346a89fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:20.742846 22777 tablet_bootstrap.cc:492] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: Bootstrap starting.
I20260808 02:24:20.744408 22777 tablet_bootstrap.cc:654] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:20.744602 22778 tablet_bootstrap.cc:492] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: Bootstrap starting.
I20260808 02:24:20.745587 22780 tablet_bootstrap.cc:492] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: Bootstrap starting.
I20260808 02:24:20.746328 22778 tablet_bootstrap.cc:654] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:20.747488 22780 tablet_bootstrap.cc:654] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:20.748639 22778 tablet_bootstrap.cc:492] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: No bootstrap required, opened a new log
I20260808 02:24:20.748826 22778 ts_tablet_manager.cc:1397] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260808 02:24:20.749707 22780 tablet_bootstrap.cc:492] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: No bootstrap required, opened a new log
I20260808 02:24:20.749874 22780 ts_tablet_manager.cc:1397] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 02:24:20.749719 22778 raft_consensus.cc:348] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.749966 22778 raft_consensus.cc:374] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:20.750072 22778 raft_consensus.cc:729] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Initialized, Role: FOLLOWER
I20260808 02:24:20.750316 22778 consensus_queue.cc:260] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.750687 22780 raft_consensus.cc:348] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.750969 22780 raft_consensus.cc:374] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:20.750861 22786 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.751031 22778 ts_tablet_manager.cc:1428] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:24:20.751083 22780 raft_consensus.cc:729] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a345ed5dae04e3fb0428e1387c81a73, State: Initialized, Role: FOLLOWER
I20260808 02:24:20.751185 22786 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:20.751971 22777 tablet_bootstrap.cc:492] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: No bootstrap required, opened a new log
I20260808 02:24:20.752136 22777 ts_tablet_manager.cc:1397] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260808 02:24:20.752969 22777 raft_consensus.cc:348] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.753216 22777 raft_consensus.cc:374] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:20.753324 22777 raft_consensus.cc:729] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b347aa5f1b8470b9d38d04cc4373637, State: Initialized, Role: FOLLOWER
I20260808 02:24:20.753669 22777 consensus_queue.cc:260] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.754072 22780 consensus_queue.cc:260] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.754726 22780 ts_tablet_manager.cc:1428] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260808 02:24:20.754551 22785 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.754997 22785 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:20.755317 22777 ts_tablet_manager.cc:1428] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:20.755199 22784 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.755585 22784 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:20.817886 22786 raft_consensus.cc:482] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:20.818137 22786 raft_consensus.cc:504] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.818845 22786 leader_election.cc:288] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:20.819442 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d642ee20bab4e259c492bb3346a89fa" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637" is_pre_election: true
I20260808 02:24:20.819617 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d642ee20bab4e259c492bb3346a89fa" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73" is_pre_election: true
I20260808 02:24:20.819761 22418 raft_consensus.cc:2454] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:20.819885 22565 raft_consensus.cc:2454] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:20.820264 22633 leader_election.cc:302] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5a345ed5dae04e3fb0428e1387c81a73, 7b347aa5f1b8470b9d38d04cc4373637, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:20.820605 22786 raft_consensus.cc:2781] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:20.820724 22786 raft_consensus.cc:482] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:20.820850 22786 raft_consensus.cc:3037] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:20.822929 22786 raft_consensus.cc:504] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.823568 22786 leader_election.cc:288] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Requested vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:20.823966 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d642ee20bab4e259c492bb3346a89fa" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637"
I20260808 02:24:20.824090 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d642ee20bab4e259c492bb3346a89fa" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73"
I20260808 02:24:20.824198 22418 raft_consensus.cc:3037] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:20.824326 22565 raft_consensus.cc:3037] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:20.827261 22418 raft_consensus.cc:2454] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:20.827391 22565 raft_consensus.cc:2454] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:20.827893 22636 leader_election.cc:302] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5a345ed5dae04e3fb0428e1387c81a73, 7b347aa5f1b8470b9d38d04cc4373637, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:20.828243 22786 raft_consensus.cc:2781] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:20.828442 22786 raft_consensus.cc:686] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Becoming Leader. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Running, Role: LEADER
I20260808 02:24:20.828738 22786 consensus_queue.cc:237] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:20.830442 22788 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: TxnStatusTablet state changed. Reason: New leader d356e1b9aa504442b4837001d5fdf804. Latest consensus state: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.830806 22788 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:24:20.831542 22801 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:24:20.831843 22801 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:24:20.832119 22250 catalog_manager.cc:5495] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 reported cstate change: term changed from 0 to 1, leader changed from <none> to d356e1b9aa504442b4837001d5fdf804 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } health_report { overall_health: HEALTHY } } }
I20260808 02:24:20.855188 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.848284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.19.231.3:46031)
0808 02:24:20.848495 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.848501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.848571 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:24:20.848836 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.848842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.848944 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.849118 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:20.849439 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.850585 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.852347 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.853081 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.853218 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.853715 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:20.854451 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:20.854643 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.854803 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.854891 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:24:20.855226 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.848259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:46031 (local address 127.0.0.1:60746)
0808 02:24:20.848505 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.848522 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.848641 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.849122 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.849126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.849135 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:24:20.849311 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.849319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.850732 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.850736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.852145 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.852178 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.853239 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.853244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.853408 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.853462 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:24:20.854592 (+ 1130us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:24:20.854674 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.854758 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.855004 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 02:24:20.863025 22565 raft_consensus.cc:1261] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:20.863127 22418 raft_consensus.cc:1261] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:20.863984 22788 consensus_queue.cc:1035] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:20.864394 22786 consensus_queue.cc:1035] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:20.877399 22785 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: TxnStatusTablet state changed. Reason: New leader d356e1b9aa504442b4837001d5fdf804. Latest consensus state: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.877719 22785 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:20.879962 22804 mvcc.cc:205] Tried to move back new op lower bound from 7316094406085373952 to 7316094405956665344. Current Snapshot: MvccSnapshot[applied={T|T < 7316094406085373952}]
I20260808 02:24:20.881083 22784 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: TxnStatusTablet state changed. Reason: New leader d356e1b9aa504442b4837001d5fdf804. Latest consensus state: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.881377 22784 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:20.881815 22805 mvcc.cc:205] Tried to move back new op lower bound from 7316094406085373952 to 7316094405956665344. Current Snapshot: MvccSnapshot[applied={T|T < 7316094406085373952}]
I20260808 02:24:20.883462 22788 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.884924 22785 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.885203 22785 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P 5a345ed5dae04e3fb0428e1387c81a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:20.886263 22788 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:24:20.889549 22811 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.889986 22811 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P d356e1b9aa504442b4837001d5fdf804: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:24:20.895207 22784 tablet_replica.cc:439] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } } }
I20260808 02:24:20.895457 22784 tablet_replica.cc:442] T 3d642ee20bab4e259c492bb3346a89fa P 7b347aa5f1b8470b9d38d04cc4373637: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:24:20.903334 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.897237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.19.231.3:46031)
0808 02:24:20.897430 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.897437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.898027 (+ 590us) server_negotiation.cc:407] Connection header received
0808 02:24:20.898091 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.898095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.898167 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.898270 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:20.898811 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.899816 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.901225 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.901699 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.901873 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.902374 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:20.902488 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:20.902641 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.902820 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.902906 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:24:20.920763 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.914970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:42281 (local address 127.19.231.3:46031)
0808 02:24:20.915119 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:20.915124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:20.915749 (+ 625us) server_negotiation.cc:407] Connection header received
0808 02:24:20.915955 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:20.915960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:20.916023 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:20.916111 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:20.916685 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.917412 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.918905 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.919364 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.919514 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.919982 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:20.920090 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:20.920190 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:20.920317 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 02:24:20.920435 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":41}
I20260808 02:24:20.920890 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:20.915036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:46031 (local address 127.19.231.3:42281)
0808 02:24:20.915650 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:20.915668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:20.915824 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:20.916120 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:20.916125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:20.916138 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:24:20.916525 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.916534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.917545 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.917551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.918771 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:20.918786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:20.919553 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:20.919560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:20.919700 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:20.919747 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:24:20.920194 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:24:20.920206 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:24:20.920300 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 02:24:20.920496 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":65,"thread_start_us":355,"threads_started":1}
I20260808 02:24:20.930058 22418 raft_consensus.cc:1261] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:20.930507 22565 raft_consensus.cc:1261] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:20.930657 22812 consensus_queue.cc:1035] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:20.931078 22786 consensus_queue.cc:1035] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:20.999505 22249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:20.999819 22249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:21.003650 22249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Sending DeleteTablet for 3 replicas of tablet 6474499fdb4a40a1873365f4d5328da7
I20260808 02:24:21.004700 22684 tablet_service.cc:1514] Processing DeleteTablet for tablet 6474499fdb4a40a1873365f4d5328da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:20 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:60742
I20260808 02:24:21.004886 22537 tablet_service.cc:1514] Processing DeleteTablet for tablet 6474499fdb4a40a1873365f4d5328da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:20 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260808 02:24:21.005218 22391 tablet_service.cc:1514] Processing DeleteTablet for tablet 6474499fdb4a40a1873365f4d5328da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:20 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43204
I20260808 02:24:21.013058 22250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:21.013995 22250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:21.014644 22823 tablet_replica.cc:330] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: stopping tablet replica
I20260808 02:24:21.015488 22823 raft_consensus.cc:2229] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:21.016521 22823 raft_consensus.cc:2258] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:21.017354 22821 tablet_replica.cc:330] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: stopping tablet replica
I20260808 02:24:21.017988 22821 raft_consensus.cc:2229] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:21.018901 22821 raft_consensus.cc:2258] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:21.019773 22822 tablet_replica.cc:330] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: stopping tablet replica
I20260808 02:24:21.020215 22822 raft_consensus.cc:2229] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:21.020597 22822 raft_consensus.cc:2258] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:21.021807 22821 ts_tablet_manager.cc:1905] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:21.023140 22822 ts_tablet_manager.cc:1905] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:21.025131 22391 tablet_service.cc:1467] Processing CreateTablet for tablet bd084318b5d4403dbf2cf483f1797eed (DEFAULT_TABLE table=test-table [id=e8231103574b4d57870177cd7b9460de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:21.025640 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd084318b5d4403dbf2cf483f1797eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:21.030046 22821 ts_tablet_manager.cc:1918] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:21.030229 22821 log.cc:1196] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/6474499fdb4a40a1873365f4d5328da7
I20260808 02:24:21.031459 22821 ts_tablet_manager.cc:1939] T 6474499fdb4a40a1873365f4d5328da7 P d356e1b9aa504442b4837001d5fdf804: Deleting consensus metadata
I20260808 02:24:21.032413 22823 ts_tablet_manager.cc:1905] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:21.033217 22684 tablet_service.cc:1467] Processing CreateTablet for tablet bd084318b5d4403dbf2cf483f1797eed (DEFAULT_TABLE table=test-table [id=e8231103574b4d57870177cd7b9460de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:21.033457 22234 catalog_manager.cc:4841] TS d356e1b9aa504442b4837001d5fdf804 (127.19.231.3:46031): tablet 6474499fdb4a40a1873365f4d5328da7 (table test-table [id=9e0579eef0ab479983ba0335b912bfc8]) successfully deleted
I20260808 02:24:21.034260 22537 tablet_service.cc:1467] Processing CreateTablet for tablet bd084318b5d4403dbf2cf483f1797eed (DEFAULT_TABLE table=test-table [id=e8231103574b4d57870177cd7b9460de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:21.034782 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd084318b5d4403dbf2cf483f1797eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:21.036060 22684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd084318b5d4403dbf2cf483f1797eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:21.040174 22823 ts_tablet_manager.cc:1918] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:21.040388 22823 log.cc:1196] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/6474499fdb4a40a1873365f4d5328da7
I20260808 02:24:21.040536 22777 tablet_bootstrap.cc:492] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: Bootstrap starting.
I20260808 02:24:21.041043 22823 ts_tablet_manager.cc:1939] T 6474499fdb4a40a1873365f4d5328da7 P 7b347aa5f1b8470b9d38d04cc4373637: Deleting consensus metadata
I20260808 02:24:21.042245 22822 ts_tablet_manager.cc:1918] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:21.042428 22822 log.cc:1196] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/6474499fdb4a40a1873365f4d5328da7
I20260808 02:24:21.042454 22777 tablet_bootstrap.cc:654] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:21.042814 22780 tablet_bootstrap.cc:492] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: Bootstrap starting.
I20260808 02:24:21.043279 22822 ts_tablet_manager.cc:1939] T 6474499fdb4a40a1873365f4d5328da7 P 5a345ed5dae04e3fb0428e1387c81a73: Deleting consensus metadata
I20260808 02:24:21.043277 22232 catalog_manager.cc:4841] TS 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557): tablet 6474499fdb4a40a1873365f4d5328da7 (table test-table [id=9e0579eef0ab479983ba0335b912bfc8]) successfully deleted
I20260808 02:24:21.044512 22780 tablet_bootstrap.cc:654] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:21.045409 22235 catalog_manager.cc:4841] TS 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549): tablet 6474499fdb4a40a1873365f4d5328da7 (table test-table [id=9e0579eef0ab479983ba0335b912bfc8]) successfully deleted
I20260808 02:24:21.048090 22777 tablet_bootstrap.cc:492] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: No bootstrap required, opened a new log
I20260808 02:24:21.048262 22777 ts_tablet_manager.cc:1397] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20260808 02:24:21.049005 22777 raft_consensus.cc:348] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.049279 22777 raft_consensus.cc:374] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:21.049368 22777 raft_consensus.cc:729] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b347aa5f1b8470b9d38d04cc4373637, State: Initialized, Role: FOLLOWER
I20260808 02:24:21.049655 22777 consensus_queue.cc:260] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.052515 22778 tablet_bootstrap.cc:492] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: Bootstrap starting.
I20260808 02:24:21.053905 22777 ts_tablet_manager.cc:1428] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260808 02:24:21.054733 22778 tablet_bootstrap.cc:654] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:21.055096 22780 tablet_bootstrap.cc:492] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: No bootstrap required, opened a new log
I20260808 02:24:21.055269 22780 ts_tablet_manager.cc:1397] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260808 02:24:21.056206 22780 raft_consensus.cc:348] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.056525 22780 raft_consensus.cc:374] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:21.056617 22780 raft_consensus.cc:729] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a345ed5dae04e3fb0428e1387c81a73, State: Initialized, Role: FOLLOWER
I20260808 02:24:21.056972 22780 consensus_queue.cc:260] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.057756 22780 ts_tablet_manager.cc:1428] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:24:21.060000 22778 tablet_bootstrap.cc:492] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: No bootstrap required, opened a new log
I20260808 02:24:21.060221 22778 ts_tablet_manager.cc:1397] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260808 02:24:21.061084 22778 raft_consensus.cc:348] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.061378 22778 raft_consensus.cc:374] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:21.061483 22778 raft_consensus.cc:729] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Initialized, Role: FOLLOWER
I20260808 02:24:21.061813 22778 consensus_queue.cc:260] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.062378 22778 ts_tablet_manager.cc:1428] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:24:21.365662 22788 raft_consensus.cc:482] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:21.365922 22788 raft_consensus.cc:504] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.366807 22788 leader_election.cc:288] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:21.367334 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd084318b5d4403dbf2cf483f1797eed" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637" is_pre_election: true
I20260808 02:24:21.367575 22418 raft_consensus.cc:2454] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:21.367660 22564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd084318b5d4403dbf2cf483f1797eed" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73" is_pre_election: true
I20260808 02:24:21.367939 22564 raft_consensus.cc:2454] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:21.368033 22633 leader_election.cc:302] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b347aa5f1b8470b9d38d04cc4373637, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:21.368511 22788 raft_consensus.cc:2781] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:21.368620 22788 raft_consensus.cc:482] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:21.368717 22788 raft_consensus.cc:3037] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:21.371459 22788 raft_consensus.cc:504] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.372596 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd084318b5d4403dbf2cf483f1797eed" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637"
I20260808 02:24:21.372628 22564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd084318b5d4403dbf2cf483f1797eed" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73"
I20260808 02:24:21.372807 22418 raft_consensus.cc:3037] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:21.372805 22564 raft_consensus.cc:3037] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:21.375816 22418 raft_consensus.cc:2454] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:21.375816 22564 raft_consensus.cc:2454] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:21.376282 22636 leader_election.cc:302] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a345ed5dae04e3fb0428e1387c81a73, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:21.376611 22811 raft_consensus.cc:2781] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:21.376672 22788 leader_election.cc:288] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Requested vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:21.376839 22811 raft_consensus.cc:686] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Becoming Leader. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Running, Role: LEADER
I20260808 02:24:21.377233 22811 consensus_queue.cc:237] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:21.380242 22250 catalog_manager.cc:5495] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 reported cstate change: term changed from 0 to 1, leader changed from <none> to d356e1b9aa504442b4837001d5fdf804 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } health_report { overall_health: HEALTHY } } }
I20260808 02:24:21.427215 22418 raft_consensus.cc:1261] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:21.427580 22564 raft_consensus.cc:1261] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:21.427804 22811 consensus_queue.cc:1035] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:21.428261 22786 consensus_queue.cc:1035] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:24:21.457718 22804 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:24:23.054991 22250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:23.055325 22250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:23.060822 22250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Sending DeleteTablet for 3 replicas of tablet bd084318b5d4403dbf2cf483f1797eed
I20260808 02:24:23.061787 22537 tablet_service.cc:1514] Processing DeleteTablet for tablet bd084318b5d4403dbf2cf483f1797eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:23 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260808 02:24:23.062009 22684 tablet_service.cc:1514] Processing DeleteTablet for tablet bd084318b5d4403dbf2cf483f1797eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:23 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:60742
I20260808 02:24:23.062008 22391 tablet_service.cc:1514] Processing DeleteTablet for tablet bd084318b5d4403dbf2cf483f1797eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:23 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43204
I20260808 02:24:23.075475 22852 tablet_replica.cc:330] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: stopping tablet replica
I20260808 02:24:23.075898 22852 raft_consensus.cc:2229] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:23.076625 22852 raft_consensus.cc:2258] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:23.078186 22852 ts_tablet_manager.cc:1905] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:23.079247 22851 tablet_replica.cc:330] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: stopping tablet replica
I20260808 02:24:23.079623 22851 raft_consensus.cc:2229] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:23.080125 22851 raft_consensus.cc:2258] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:23.080260 22853 tablet_replica.cc:330] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: stopping tablet replica
I20260808 02:24:23.080513 22853 raft_consensus.cc:2229] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:23.080845 22853 raft_consensus.cc:2258] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:23.081971 22853 ts_tablet_manager.cc:1905] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:23.083407 22851 ts_tablet_manager.cc:1905] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:23.089711 22852 ts_tablet_manager.cc:1918] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:24:23.090019 22852 log.cc:1196] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/bd084318b5d4403dbf2cf483f1797eed
I20260808 02:24:23.090977 22852 ts_tablet_manager.cc:1939] T bd084318b5d4403dbf2cf483f1797eed P d356e1b9aa504442b4837001d5fdf804: Deleting consensus metadata
I20260808 02:24:23.092517 22234 catalog_manager.cc:4841] TS d356e1b9aa504442b4837001d5fdf804 (127.19.231.3:46031): tablet bd084318b5d4403dbf2cf483f1797eed (table test-table [id=e8231103574b4d57870177cd7b9460de]) successfully deleted
I20260808 02:24:23.093137 22853 ts_tablet_manager.cc:1918] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:24:23.093307 22853 log.cc:1196] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/bd084318b5d4403dbf2cf483f1797eed
I20260808 02:24:23.094141 22853 ts_tablet_manager.cc:1939] T bd084318b5d4403dbf2cf483f1797eed P 5a345ed5dae04e3fb0428e1387c81a73: Deleting consensus metadata
I20260808 02:24:23.094156 22851 ts_tablet_manager.cc:1918] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:24:23.094412 22851 log.cc:1196] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/bd084318b5d4403dbf2cf483f1797eed
I20260808 02:24:23.095084 22851 ts_tablet_manager.cc:1939] T bd084318b5d4403dbf2cf483f1797eed P 7b347aa5f1b8470b9d38d04cc4373637: Deleting consensus metadata
I20260808 02:24:23.095474 22235 catalog_manager.cc:4841] TS 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549): tablet bd084318b5d4403dbf2cf483f1797eed (table test-table [id=e8231103574b4d57870177cd7b9460de]) successfully deleted
I20260808 02:24:23.096387 22232 catalog_manager.cc:4841] TS 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557): tablet bd084318b5d4403dbf2cf483f1797eed (table test-table [id=e8231103574b4d57870177cd7b9460de]) successfully deleted
Aug 08 02:24:23 dist-test-slave-pgkr krb5kdc[22200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:24:23 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:23.142349 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:23.124126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.19.231.62:37327)
0808 02:24:23.124812 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:23.124821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:23.124934 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:24:23.125098 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:23.125103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:23.125169 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:23.125271 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:23.126826 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.127604 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.128846 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.129401 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.129552 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:23.134571 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:23.134577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:23.134582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:23.134687 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:23.138849 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:23.139605 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:23.139610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:23.139614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:23.139855 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:23.140318 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:23.140324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:23.140328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:23.140707 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:23.141106 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:23.141424 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 02:24:23.141745 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":334,"threads_started":1}
I20260808 02:24:23.146771 22250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:23.147538 22250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:23.155048 22537 tablet_service.cc:1467] Processing CreateTablet for tablet f2e8b8f1d3884580a7847e29d97b40ad (DEFAULT_TABLE table=test-table [id=9926278197384a73a618abd5939b897c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:23.155109 22391 tablet_service.cc:1467] Processing CreateTablet for tablet f2e8b8f1d3884580a7847e29d97b40ad (DEFAULT_TABLE table=test-table [id=9926278197384a73a618abd5939b897c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:23.155577 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e8b8f1d3884580a7847e29d97b40ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:23.155659 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e8b8f1d3884580a7847e29d97b40ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:23.155712 22684 tablet_service.cc:1467] Processing CreateTablet for tablet f2e8b8f1d3884580a7847e29d97b40ad (DEFAULT_TABLE table=test-table [id=9926278197384a73a618abd5939b897c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:23.156212 22684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e8b8f1d3884580a7847e29d97b40ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:23.167292 22862 tablet_bootstrap.cc:492] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: Bootstrap starting.
I20260808 02:24:23.168946 22863 tablet_bootstrap.cc:492] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: Bootstrap starting.
I20260808 02:24:23.170557 22862 tablet_bootstrap.cc:654] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:23.171509 22863 tablet_bootstrap.cc:654] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:23.174556 22864 tablet_bootstrap.cc:492] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: Bootstrap starting.
I20260808 02:24:23.175294 22862 tablet_bootstrap.cc:492] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: No bootstrap required, opened a new log
I20260808 02:24:23.175566 22862 ts_tablet_manager.cc:1397] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260808 02:24:23.177630 22864 tablet_bootstrap.cc:654] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:23.177366 22862 raft_consensus.cc:348] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.177878 22862 raft_consensus.cc:374] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:23.178115 22862 raft_consensus.cc:729] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Initialized, Role: FOLLOWER
I20260808 02:24:23.178635 22862 consensus_queue.cc:260] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.179677 22862 ts_tablet_manager.cc:1428] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260808 02:24:23.182986 22863 tablet_bootstrap.cc:492] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: No bootstrap required, opened a new log
I20260808 02:24:23.183243 22863 ts_tablet_manager.cc:1397] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260808 02:24:23.183449 22864 tablet_bootstrap.cc:492] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: No bootstrap required, opened a new log
I20260808 02:24:23.183988 22864 ts_tablet_manager.cc:1397] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260808 02:24:23.185195 22863 raft_consensus.cc:348] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.185650 22863 raft_consensus.cc:374] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:23.185901 22863 raft_consensus.cc:729] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a345ed5dae04e3fb0428e1387c81a73, State: Initialized, Role: FOLLOWER
I20260808 02:24:23.185699 22864 raft_consensus.cc:348] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.186255 22864 raft_consensus.cc:374] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:23.186497 22864 raft_consensus.cc:729] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b347aa5f1b8470b9d38d04cc4373637, State: Initialized, Role: FOLLOWER
I20260808 02:24:23.186306 22863 consensus_queue.cc:260] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.186933 22864 consensus_queue.cc:260] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.188102 22863 ts_tablet_manager.cc:1428] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:23.188583 22864 ts_tablet_manager.cc:1428] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260808 02:24:23.226747 22869 raft_consensus.cc:482] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:23.227365 22869 raft_consensus.cc:504] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.229598 22869 leader_election.cc:288] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549), d356e1b9aa504442b4837001d5fdf804 (127.19.231.3:46031)
I20260808 02:24:23.230850 22833 raft_consensus.cc:482] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:23.231313 22833 raft_consensus.cc:504] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.232427 22833 leader_election.cc:288] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:23.233798 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e8b8f1d3884580a7847e29d97b40ad" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637" is_pre_election: true
I20260808 02:24:23.234192 22418 raft_consensus.cc:2454] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:23.235308 22633 leader_election.cc:302] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b347aa5f1b8470b9d38d04cc4373637, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:23.235845 22833 raft_consensus.cc:2781] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:23.236029 22833 raft_consensus.cc:482] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:23.236232 22833 raft_consensus.cc:3037] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:23.243364 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e8b8f1d3884580a7847e29d97b40ad" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73" is_pre_election: true
I20260808 02:24:23.243824 22565 raft_consensus.cc:2454] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:23.247143 22833 raft_consensus.cc:504] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.249089 22418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e8b8f1d3884580a7847e29d97b40ad" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637"
I20260808 02:24:23.249460 22418 raft_consensus.cc:3037] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:23.250730 22833 leader_election.cc:288] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Requested vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:23.252712 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e8b8f1d3884580a7847e29d97b40ad" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73"
I20260808 02:24:23.253093 22565 raft_consensus.cc:3037] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:23.256953 22418 raft_consensus.cc:2454] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:23.259194 22633 leader_election.cc:302] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b347aa5f1b8470b9d38d04cc4373637, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:23.259991 22833 raft_consensus.cc:2781] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:23.260643 22833 raft_consensus.cc:686] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Becoming Leader. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Running, Role: LEADER
I20260808 02:24:23.261272 22833 consensus_queue.cc:237] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.262888 22565 raft_consensus.cc:2454] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:23.268795 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:23.230084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:33443 (local address 127.19.231.3:46031)
0808 02:24:23.246397 (+ 16313us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:23.246405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:23.249222 (+ 2817us) server_negotiation.cc:407] Connection header received
0808 02:24:23.249299 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:23.249304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:23.249407 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:23.250075 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:23.255754 (+ 5679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.257917 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.262159 (+ 4242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.264691 (+ 2532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.265057 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:23.265192 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:23.267763 (+ 2571us) server_negotiation.cc:299] Negotiation successful
0808 02:24:23.268013 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16013,"thread_start_us":20043,"threads_started":1}
I20260808 02:24:23.268345 22250 catalog_manager.cc:5495] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 reported cstate change: term changed from 0 to 1, leader changed from <none> to d356e1b9aa504442b4837001d5fdf804 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } health_report { overall_health: HEALTHY } } }
I20260808 02:24:23.269249 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:23.230410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:46031 (local address 127.19.231.1:33443)
0808 02:24:23.248868 (+ 18458us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:23.248890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:23.249089 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:23.249843 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:23.249851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:23.249887 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:23.255390 (+ 5503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:23.255407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.258228 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.258236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:23.261906 (+ 3670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:23.261923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.266835 (+ 4912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.266848 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:23.267387 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:23.267436 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:24:23.267633 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 02:24:23.267913 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18238,"mutex_wait_us":1,"thread_start_us":17769,"threads_started":1}
I20260808 02:24:23.270717 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e8b8f1d3884580a7847e29d97b40ad" candidate_uuid: "7b347aa5f1b8470b9d38d04cc4373637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d356e1b9aa504442b4837001d5fdf804" is_pre_election: true
I20260808 02:24:23.276784 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:23.230166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:34571 (local address 127.19.231.2:39549)
0808 02:24:23.243413 (+ 13247us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:23.243424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:23.248574 (+ 5150us) server_negotiation.cc:407] Connection header received
0808 02:24:23.251981 (+ 3407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:23.251988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:23.252071 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:23.252532 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:23.258132 (+ 5600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.264972 (+ 6840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.272577 (+ 7605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.274981 (+ 2404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.275290 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:23.275376 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:23.275926 (+ 550us) server_negotiation.cc:299] Negotiation successful
0808 02:24:23.276131 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12960,"thread_start_us":11000,"threads_started":1}
I20260808 02:24:23.276856 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:23.229868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:39549 (local address 127.19.231.1:34571)
0808 02:24:23.247047 (+ 17179us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:23.247078 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:23.247327 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:23.252396 (+ 5069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:23.252401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:23.252436 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:23.254930 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:23.254952 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.265258 (+ 10306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.265265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:23.272309 (+ 7044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:23.272334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.275408 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.275419 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:23.275682 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:23.275718 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:24:23.275808 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:24:23.275949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16952,"mutex_wait_us":2700,"thread_start_us":17609,"threads_started":1}
I20260808 02:24:23.277663 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e8b8f1d3884580a7847e29d97b40ad" candidate_uuid: "7b347aa5f1b8470b9d38d04cc4373637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73" is_pre_election: true
I20260808 02:24:23.278115 22565 raft_consensus.cc:2379] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b347aa5f1b8470b9d38d04cc4373637 in current term 1: Already voted for candidate d356e1b9aa504442b4837001d5fdf804 in this term.
I20260808 02:24:23.279225 22342 leader_election.cc:302] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b347aa5f1b8470b9d38d04cc4373637; no voters: 5a345ed5dae04e3fb0428e1387c81a73, d356e1b9aa504442b4837001d5fdf804
I20260808 02:24:23.280166 22869 raft_consensus.cc:2726] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 02:24:23.310403 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:23.305378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.19.231.3:46031)
0808 02:24:23.305566 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:23.305572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:23.305592 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:24:23.305648 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:23.305651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:23.305716 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:23.305842 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:23.306261 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.307162 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.308627 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:23.309073 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:23.309219 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:23.309663 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:23.309769 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:23.309878 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:23.310037 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 02:24:23.310134 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:24:23.321727 22418 raft_consensus.cc:1261] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:23.322453 22565 raft_consensus.cc:1261] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:23.322399 22833 consensus_queue.cc:1035] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:23.323145 22833 consensus_queue.cc:1035] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:23.379675 22249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:23.379997 22249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:23.383531 22249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Sending DeleteTablet for 3 replicas of tablet f2e8b8f1d3884580a7847e29d97b40ad
I20260808 02:24:23.384616 22391 tablet_service.cc:1514] Processing DeleteTablet for tablet f2e8b8f1d3884580a7847e29d97b40ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:23 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43204
I20260808 02:24:23.384861 22537 tablet_service.cc:1514] Processing DeleteTablet for tablet f2e8b8f1d3884580a7847e29d97b40ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:23 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260808 02:24:23.384990 22685 tablet_service.cc:1514] Processing DeleteTablet for tablet f2e8b8f1d3884580a7847e29d97b40ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:23 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:60742
I20260808 02:24:23.385063 22851 tablet_replica.cc:330] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: stopping tablet replica
I20260808 02:24:23.385314 22853 tablet_replica.cc:330] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: stopping tablet replica
I20260808 02:24:23.385337 22852 tablet_replica.cc:330] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: stopping tablet replica
I20260808 02:24:23.385432 22851 raft_consensus.cc:2229] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:23.385660 22852 raft_consensus.cc:2229] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:23.385628 22853 raft_consensus.cc:2229] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:23.385828 22851 raft_consensus.cc:2258] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:23.385962 22853 raft_consensus.cc:2258] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:23.386137 22852 raft_consensus.cc:2258] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:23.386116 22249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:23.387183 22249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:23.387305 22851 ts_tablet_manager.cc:1905] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:23.387324 22852 ts_tablet_manager.cc:1905] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:23.387617 22853 ts_tablet_manager.cc:1905] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:23.396440 22853 ts_tablet_manager.cc:1918] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:23.396390 22391 tablet_service.cc:1467] Processing CreateTablet for tablet c5557735d1934ec29c57dd1031aa0f3b (DEFAULT_TABLE table=test-table [id=76692dd50df54a5399430f85f5c9a4c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:23.396799 22853 log.cc:1196] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/f2e8b8f1d3884580a7847e29d97b40ad
I20260808 02:24:23.397070 22685 tablet_service.cc:1467] Processing CreateTablet for tablet c5557735d1934ec29c57dd1031aa0f3b (DEFAULT_TABLE table=test-table [id=76692dd50df54a5399430f85f5c9a4c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:23.397256 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5557735d1934ec29c57dd1031aa0f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:23.397313 22537 tablet_service.cc:1467] Processing CreateTablet for tablet c5557735d1934ec29c57dd1031aa0f3b (DEFAULT_TABLE table=test-table [id=76692dd50df54a5399430f85f5c9a4c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:23.397625 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5557735d1934ec29c57dd1031aa0f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:23.397663 22853 ts_tablet_manager.cc:1939] T f2e8b8f1d3884580a7847e29d97b40ad P 5a345ed5dae04e3fb0428e1387c81a73: Deleting consensus metadata
I20260808 02:24:23.397980 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5557735d1934ec29c57dd1031aa0f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:23.399070 22235 catalog_manager.cc:4841] TS 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549): tablet f2e8b8f1d3884580a7847e29d97b40ad (table test-table [id=9926278197384a73a618abd5939b897c]) successfully deleted
I20260808 02:24:23.403903 22862 tablet_bootstrap.cc:492] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: Bootstrap starting.
I20260808 02:24:23.407260 22852 ts_tablet_manager.cc:1918] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:23.407384 22863 tablet_bootstrap.cc:492] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: Bootstrap starting.
I20260808 02:24:23.407850 22852 log.cc:1196] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/f2e8b8f1d3884580a7847e29d97b40ad
I20260808 02:24:23.407980 22862 tablet_bootstrap.cc:654] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:23.408216 22851 ts_tablet_manager.cc:1918] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:24:23.408545 22851 log.cc:1196] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/f2e8b8f1d3884580a7847e29d97b40ad
I20260808 02:24:23.408954 22852 ts_tablet_manager.cc:1939] T f2e8b8f1d3884580a7847e29d97b40ad P d356e1b9aa504442b4837001d5fdf804: Deleting consensus metadata
I20260808 02:24:23.409224 22851 ts_tablet_manager.cc:1939] T f2e8b8f1d3884580a7847e29d97b40ad P 7b347aa5f1b8470b9d38d04cc4373637: Deleting consensus metadata
I20260808 02:24:23.410579 22863 tablet_bootstrap.cc:654] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:23.410701 22864 tablet_bootstrap.cc:492] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: Bootstrap starting.
I20260808 02:24:23.410666 22234 catalog_manager.cc:4841] TS d356e1b9aa504442b4837001d5fdf804 (127.19.231.3:46031): tablet f2e8b8f1d3884580a7847e29d97b40ad (table test-table [id=9926278197384a73a618abd5939b897c]) successfully deleted
I20260808 02:24:23.412045 22232 catalog_manager.cc:4841] TS 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557): tablet f2e8b8f1d3884580a7847e29d97b40ad (table test-table [id=9926278197384a73a618abd5939b897c]) successfully deleted
I20260808 02:24:23.413664 22864 tablet_bootstrap.cc:654] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:23.414139 22863 tablet_bootstrap.cc:492] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: No bootstrap required, opened a new log
I20260808 02:24:23.414438 22863 ts_tablet_manager.cc:1397] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260808 02:24:23.415679 22863 raft_consensus.cc:348] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.416184 22863 raft_consensus.cc:374] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:23.416374 22863 raft_consensus.cc:729] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a345ed5dae04e3fb0428e1387c81a73, State: Initialized, Role: FOLLOWER
I20260808 02:24:23.416944 22863 consensus_queue.cc:260] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.418946 22863 ts_tablet_manager.cc:1428] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260808 02:24:23.422149 22862 tablet_bootstrap.cc:492] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: No bootstrap required, opened a new log
I20260808 02:24:23.422395 22862 ts_tablet_manager.cc:1397] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260808 02:24:23.422564 22864 tablet_bootstrap.cc:492] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: No bootstrap required, opened a new log
I20260808 02:24:23.422851 22864 ts_tablet_manager.cc:1397] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20260808 02:24:23.423712 22862 raft_consensus.cc:348] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.423868 22864 raft_consensus.cc:348] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.424114 22862 raft_consensus.cc:374] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:23.424152 22864 raft_consensus.cc:374] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:23.424285 22864 raft_consensus.cc:729] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b347aa5f1b8470b9d38d04cc4373637, State: Initialized, Role: FOLLOWER
I20260808 02:24:23.424283 22862 raft_consensus.cc:729] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Initialized, Role: FOLLOWER
I20260808 02:24:23.424701 22862 consensus_queue.cc:260] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.424631 22864 consensus_queue.cc:260] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.425345 22862 ts_tablet_manager.cc:1428] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 02:24:23.425441 22864 ts_tablet_manager.cc:1428] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 02:24:23.718024 22838 raft_consensus.cc:482] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:23.718276 22838 raft_consensus.cc:504] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.719112 22838 leader_election.cc:288] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:23.719714 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5557735d1934ec29c57dd1031aa0f3b" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73" is_pre_election: true
I20260808 02:24:23.719725 22417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5557735d1934ec29c57dd1031aa0f3b" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637" is_pre_election: true
I20260808 02:24:23.720000 22417 raft_consensus.cc:2454] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:23.719990 22565 raft_consensus.cc:2454] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 0.
I20260808 02:24:23.720472 22633 leader_election.cc:302] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b347aa5f1b8470b9d38d04cc4373637, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:23.720781 22838 raft_consensus.cc:2781] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:23.720902 22838 raft_consensus.cc:482] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:23.721019 22838 raft_consensus.cc:3037] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:23.723644 22838 raft_consensus.cc:504] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.724267 22838 leader_election.cc:288] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Requested vote from peers 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557), 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549)
I20260808 02:24:23.724857 22417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5557735d1934ec29c57dd1031aa0f3b" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b347aa5f1b8470b9d38d04cc4373637"
I20260808 02:24:23.724936 22565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5557735d1934ec29c57dd1031aa0f3b" candidate_uuid: "d356e1b9aa504442b4837001d5fdf804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a345ed5dae04e3fb0428e1387c81a73"
I20260808 02:24:23.725080 22417 raft_consensus.cc:3037] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:23.725229 22565 raft_consensus.cc:3037] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:23.728343 22417 raft_consensus.cc:2454] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:23.728826 22633 leader_election.cc:302] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b347aa5f1b8470b9d38d04cc4373637, d356e1b9aa504442b4837001d5fdf804; no voters:
I20260808 02:24:23.729209 22838 raft_consensus.cc:2781] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:23.729439 22838 raft_consensus.cc:686] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Becoming Leader. State: Replica: d356e1b9aa504442b4837001d5fdf804, State: Running, Role: LEADER
I20260808 02:24:23.729820 22838 consensus_queue.cc:237] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } }
I20260808 02:24:23.730793 22565 raft_consensus.cc:2454] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d356e1b9aa504442b4837001d5fdf804 in term 1.
I20260808 02:24:23.733846 22249 catalog_manager.cc:5495] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 reported cstate change: term changed from 0 to 1, leader changed from <none> to d356e1b9aa504442b4837001d5fdf804 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "d356e1b9aa504442b4837001d5fdf804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d356e1b9aa504442b4837001d5fdf804" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 46031 } health_report { overall_health: HEALTHY } } }
I20260808 02:24:23.796831 22417 raft_consensus.cc:1261] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:23.797331 22564 raft_consensus.cc:1261] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Refusing update from remote peer d356e1b9aa504442b4837001d5fdf804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:23.797369 22833 consensus_queue.cc:1035] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b347aa5f1b8470b9d38d04cc4373637" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:23.797950 22833 consensus_queue.cc:1035] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a345ed5dae04e3fb0428e1387c81a73" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:23.803375 22835 mvcc.cc:205] Tried to move back new op lower bound from 7316094418107506688 to 7316094417840005120. Current Snapshot: MvccSnapshot[applied={T|T < 7316094418107506688}]
W20260808 02:24:23.820318 22831 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 02:24:25.413826 22249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:25.414106 22249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:25.419878 22249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95b0c5ea53e54cea878e0c94db41866a: Sending DeleteTablet for 3 replicas of tablet c5557735d1934ec29c57dd1031aa0f3b
I20260808 02:24:25.421020 22391 tablet_service.cc:1514] Processing DeleteTablet for tablet c5557735d1934ec29c57dd1031aa0f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:25 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43204
I20260808 02:24:25.421177 22684 tablet_service.cc:1514] Processing DeleteTablet for tablet c5557735d1934ec29c57dd1031aa0f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:25 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:60742
I20260808 02:24:25.421205 22537 tablet_service.cc:1514] Processing DeleteTablet for tablet c5557735d1934ec29c57dd1031aa0f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:25 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260808 02:24:25.423120 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22322
I20260808 02:24:25.425109 22911 tablet_replica.cc:330] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: stopping tablet replica
I20260808 02:24:25.425510 22911 raft_consensus.cc:2229] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:25.425980 22911 raft_consensus.cc:2258] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:25.426067 22910 tablet_replica.cc:330] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: stopping tablet replica
I20260808 02:24:25.426338 22910 raft_consensus.cc:2229] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:25.426852 22910 raft_consensus.cc:2258] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:25.427595 22911 ts_tablet_manager.cc:1905] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:25.428071 22909 tablet_replica.cc:330] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: stopping tablet replica
I20260808 02:24:25.428370 22909 raft_consensus.cc:2229] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:25.428977 22909 raft_consensus.cc:2258] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:25.430502 22910 ts_tablet_manager.cc:1905] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:25.430552 22909 ts_tablet_manager.cc:1905] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:25.436677 22911 ts_tablet_manager.cc:1918] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:24:25.436968 22911 log.cc:1196] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/c5557735d1934ec29c57dd1031aa0f3b
I20260808 02:24:25.437793 22911 ts_tablet_manager.cc:1939] T c5557735d1934ec29c57dd1031aa0f3b P 5a345ed5dae04e3fb0428e1387c81a73: Deleting consensus metadata
I20260808 02:24:25.439162 22910 ts_tablet_manager.cc:1918] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:24:25.439271 22909 ts_tablet_manager.cc:1918] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 02:24:25.439342 22910 log.cc:1196] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/c5557735d1934ec29c57dd1031aa0f3b
I20260808 02:24:25.439463 22909 log.cc:1196] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/c5557735d1934ec29c57dd1031aa0f3b
I20260808 02:24:25.439745 22235 catalog_manager.cc:4841] TS 5a345ed5dae04e3fb0428e1387c81a73 (127.19.231.2:39549): tablet c5557735d1934ec29c57dd1031aa0f3b (table test-table [id=76692dd50df54a5399430f85f5c9a4c2]) successfully deleted
I20260808 02:24:25.439973 22909 ts_tablet_manager.cc:1939] T c5557735d1934ec29c57dd1031aa0f3b P 7b347aa5f1b8470b9d38d04cc4373637: Deleting consensus metadata
I20260808 02:24:25.440155 22910 ts_tablet_manager.cc:1939] T c5557735d1934ec29c57dd1031aa0f3b P d356e1b9aa504442b4837001d5fdf804: Deleting consensus metadata
I20260808 02:24:25.441182 22232 catalog_manager.cc:4841] TS 7b347aa5f1b8470b9d38d04cc4373637 (127.19.231.1:46557): tablet c5557735d1934ec29c57dd1031aa0f3b (table test-table [id=76692dd50df54a5399430f85f5c9a4c2]) successfully deleted
I20260808 02:24:25.441278 22234 catalog_manager.cc:4841] TS d356e1b9aa504442b4837001d5fdf804 (127.19.231.3:46031): tablet c5557735d1934ec29c57dd1031aa0f3b (table test-table [id=76692dd50df54a5399430f85f5c9a4c2]) successfully deleted
Aug 08 02:24:25 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:25.455441 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:25.425544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.19.231.1:46557)
0808 02:24:25.429677 (+ 4133us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:25.429686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:25.429724 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:24:25.429796 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:25.429803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:25.429893 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:25.432546 (+ 2653us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:25.432858 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:25.434300 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:25.436565 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:25.437413 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:25.438238 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:25.447383 (+ 9145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:25.447401 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:25.447418 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:25.447449 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:25.451978 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:25.452791 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:25.452798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:25.452801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:25.453009 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:25.453503 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:25.453509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:25.453512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:25.453940 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:25.454262 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:25.454686 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 02:24:25.454950 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3850,"thread_start_us":3585,"threads_started":1}
W20260808 02:24:25.456261 22460 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37532
W20260808 02:24:25.457082 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22322: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:25.457312 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 22322
I20260808 02:24:25.480700 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22469
Aug 08 02:24:25 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:25.497622 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:25.481548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.19.231.2:39549)
0808 02:24:25.482301 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:25.482306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:25.482325 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:24:25.482389 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:25.482393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:25.482460 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:25.482568 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:25.484083 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:25.485113 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:25.486328 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:25.486998 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:25.487167 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:25.491448 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:25.491462 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:25.491478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:25.491513 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:25.494459 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:25.495310 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:25.495317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:25.495319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:25.495456 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:25.495894 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:25.495898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:25.495900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:25.496255 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:25.496609 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:25.496984 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 02:24:25.497282 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":359,"threads_started":1}
W20260808 02:24:25.498279 22605 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39016
W20260808 02:24:25.499017 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:25.499174 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 22469
I20260808 02:24:25.522094 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22615
Aug 08 02:24:25 dist-test-slave-pgkr krb5kdc[22200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:25.538030 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:25.523063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.19.231.3:46031)
0808 02:24:25.523635 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:25.523640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:25.523659 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:24:25.523720 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:25.523724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:25.523788 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:25.523895 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:25.525217 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:25.525978 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:25.527282 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:25.527752 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:25.527864 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:25.532349 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:25.532364 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:25.532381 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:25.532421 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:25.535260 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:25.535912 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:25.535920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:25.535922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:25.536061 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:25.536450 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:25.536456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:25.536459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:25.536833 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:25.537148 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:25.537431 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 02:24:25.537692 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":281,"threads_started":1}
W20260808 02:24:25.538794 22753 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33244
W20260808 02:24:25.539425 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22615: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:25.539615 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 22615
I20260808 02:24:25.570873 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22216
I20260808 02:24:25.572279 22291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:25.695981 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 22216
2026-08-08T02:24:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8973 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:25 dist-test-slave-pgkr krb5kdc[22919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:25 dist-test-slave-pgkr krb5kdc[22919](info): set up 2 sockets
Aug 08 02:24:25 dist-test-slave-pgkr krb5kdc[22919](info): commencing operation
krb5kdc: starting...
W20260808 02:24:26.832474 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:26 dist-test-slave-pgkr krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:27.054337 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41745
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:27.395848 22933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:27.396214 22933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:27.396394 22933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:27.404439 22933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:27.404600 22933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:27.404709 22933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:27.404819 22933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:27.404904 22933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:27.415050 22933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:27.416447 22933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:27.417847 22933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:27.430109 22939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:27.430195 22942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:27.430200 22940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:27.431412 22933 server_base.cc:1034] running on GCE node
I20260808 02:24:27.432328 22933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:27.433832 22933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:27.435129 22933 hybrid_clock.cc:648] HybridClock initialized: now 1786155867435025 us; error 50 us; skew 500 ppm
I20260808 02:24:27.435515 22933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:27 dist-test-slave-pgkr krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:27.438859 22933 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:27.440357 22933 webserver.cc:460] Webserver started at http://127.19.231.62:32951/ using document root <none> and password file <none>
I20260808 02:24:27.441103 22933 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:27.441242 22933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:27.441555 22933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:27.444629 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "0078e20b7bbf4d89bed83f51d8579479"
format_stamp: "Formatted at 2026-08-08 02:24:27 on dist-test-slave-pgkr"
server_key: "3e5bec2ec8989782ad989e2d8407540f"
server_key_iv: "db3886a4d9c7a7693f1c29ee8e95b680"
server_key_version: "encryptionkey@0"
I20260808 02:24:27.445403 22933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "0078e20b7bbf4d89bed83f51d8579479"
format_stamp: "Formatted at 2026-08-08 02:24:27 on dist-test-slave-pgkr"
server_key: "3e5bec2ec8989782ad989e2d8407540f"
server_key_iv: "db3886a4d9c7a7693f1c29ee8e95b680"
server_key_version: "encryptionkey@0"
I20260808 02:24:27.450888 22933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:24:27.454813 22949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:27.456187 22933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:27.456375 22933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "0078e20b7bbf4d89bed83f51d8579479"
format_stamp: "Formatted at 2026-08-08 02:24:27 on dist-test-slave-pgkr"
server_key: "3e5bec2ec8989782ad989e2d8407540f"
server_key_iv: "db3886a4d9c7a7693f1c29ee8e95b680"
server_key_version: "encryptionkey@0"
I20260808 02:24:27.456598 22933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:27.495797 22933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:27.500186 22933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:27.500483 22933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:27.519683 22933 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:41745
I20260808 02:24:27.519707 23002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:41745 every 8 connection(s)
I20260808 02:24:27.521152 22933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:27.525027 23003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:27.527042 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 22933
I20260808 02:24:27.527281 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:27.527930 20380 external_mini_cluster.cc:1428] Setting key 1471c604e2b2bda887b2b407ae2d7e25
I20260808 02:24:27.537124 23003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479: Bootstrap starting.
I20260808 02:24:27.541080 23003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:27.542399 23003 log.cc:824] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:24:27 dist-test-slave-pgkr krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:27.546056 23003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479: No bootstrap required, opened a new log
I20260808 02:24:27.552709 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:27.529907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.19.231.62:41745)
0808 02:24:27.530753 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:27.530765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:27.530807 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:24:27.532494 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:27.532516 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:27.532881 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:27.533211 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:27.535164 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:27.536443 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:27.538179 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:27.538920 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:27.539055 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:27.544130 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:27.544141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:27.544160 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:27.544207 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:27.548139 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:27.548824 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:27.548833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:27.548841 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:27.549047 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:27.549411 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:27.549416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:27.549420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:27.550165 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:27.550457 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:27.550807 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 02:24:27.551246 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":406,"threads_started":1}
I20260808 02:24:27.556684 23003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0078e20b7bbf4d89bed83f51d8579479" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41745 } }
I20260808 02:24:27.556988 23003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:27.557060 23003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0078e20b7bbf4d89bed83f51d8579479, State: Initialized, Role: FOLLOWER
I20260808 02:24:27.557529 23003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0078e20b7bbf4d89bed83f51d8579479" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41745 } }
I20260808 02:24:27.557755 23003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:27.557849 23003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:27.557981 23003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:27.560230 23003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0078e20b7bbf4d89bed83f51d8579479" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41745 } }
I20260808 02:24:27.560777 23003 leader_election.cc:302] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0078e20b7bbf4d89bed83f51d8579479; no voters:
I20260808 02:24:27.561424 23003 leader_election.cc:288] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:27.561578 23008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:27.562238 23008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [term 1 LEADER]: Becoming Leader. State: Replica: 0078e20b7bbf4d89bed83f51d8579479, State: Running, Role: LEADER
I20260808 02:24:27.562780 23008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0078e20b7bbf4d89bed83f51d8579479" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41745 } }
I20260808 02:24:27.563555 23003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:27.565519 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0078e20b7bbf4d89bed83f51d8579479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0078e20b7bbf4d89bed83f51d8579479" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41745 } } }
I20260808 02:24:27.565922 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:27.566659 23010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0078e20b7bbf4d89bed83f51d8579479. Latest consensus state: current_term: 1 leader_uuid: "0078e20b7bbf4d89bed83f51d8579479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0078e20b7bbf4d89bed83f51d8579479" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41745 } } }
I20260808 02:24:27.567793 23010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:27.566804 23013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:27.572574 23013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:27.581655 23013 catalog_manager.cc:1347] Generated new cluster ID: 6e8bb4925b5242eaa6c893d291ac26de
I20260808 02:24:27.581786 23013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:27.593777 23013 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:27.595095 23013 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:27.605710 23013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0078e20b7bbf4d89bed83f51d8579479: Generated new TSK 0
I20260808 02:24:27.606478 23013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:27.661911 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41745
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:28.010278 23030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:28.010562 23030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:28.010743 23030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:28.018136 23030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:28.018241 23030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:28.018429 23030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:28.028508 23030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:28.029917 23030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:28.031418 23030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:28.044906 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:28.044960 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:28.045521 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:28.045840 23030 server_base.cc:1034] running on GCE node
I20260808 02:24:28.046480 23030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:28.047477 23030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:28.048683 23030 hybrid_clock.cc:648] HybridClock initialized: now 1786155868048641 us; error 35 us; skew 500 ppm
I20260808 02:24:28.049005 23030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:28 dist-test-slave-pgkr krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:28.052336 23030 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:28.053839 23030 webserver.cc:460] Webserver started at http://127.19.231.1:45973/ using document root <none> and password file <none>
I20260808 02:24:28.054508 23030 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:28.054665 23030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:28.055011 23030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:28.058009 23030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "498016ba9e864791bd0e14e7422de2e9"
format_stamp: "Formatted at 2026-08-08 02:24:28 on dist-test-slave-pgkr"
server_key: "b15d6a3a83e8c3d41319fbb059ae5f7f"
server_key_iv: "705f085ecde5561bffaab9587cc3b9ef"
server_key_version: "encryptionkey@0"
I20260808 02:24:28.058878 23030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "498016ba9e864791bd0e14e7422de2e9"
format_stamp: "Formatted at 2026-08-08 02:24:28 on dist-test-slave-pgkr"
server_key: "b15d6a3a83e8c3d41319fbb059ae5f7f"
server_key_iv: "705f085ecde5561bffaab9587cc3b9ef"
server_key_version: "encryptionkey@0"
I20260808 02:24:28.064446 23030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:28.068280 23047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:28.069687 23030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:24:28.069842 23030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "498016ba9e864791bd0e14e7422de2e9"
format_stamp: "Formatted at 2026-08-08 02:24:28 on dist-test-slave-pgkr"
server_key: "b15d6a3a83e8c3d41319fbb059ae5f7f"
server_key_iv: "705f085ecde5561bffaab9587cc3b9ef"
server_key_version: "encryptionkey@0"
I20260808 02:24:28.070031 23030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:28.092355 23030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:28.096664 23030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:28.096926 23030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:28.097875 23030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:28.099403 23030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:28.099483 23030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:28.099571 23030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:28.099611 23030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:28.135053 23030 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:45695
I20260808 02:24:28.135170 23168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:45695 every 8 connection(s)
I20260808 02:24:28.136407 23030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:28.137615 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23030
I20260808 02:24:28.137854 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:28.138712 20380 external_mini_cluster.cc:1428] Setting key 9b774010a9c2e9fe3933d19a73847555
Aug 08 02:24:28 dist-test-slave-pgkr krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:28.159197 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:28.139454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:48245 (local address 127.19.231.62:41745)
0808 02:24:28.139947 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:28.139953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:28.141260 (+ 1307us) server_negotiation.cc:407] Connection header received
0808 02:24:28.143285 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:28.143294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:28.143413 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:28.143534 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:28.145160 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.146299 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.148076 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.148764 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.148864 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:28.153135 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:28.153140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:28.153144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:28.153177 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:28.155975 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:28.156725 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:28.156729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:28.156731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:28.156852 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:28.157313 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:28.157317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:28.157319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:28.157627 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:28.157789 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:28.158550 (+ 761us) server_negotiation.cc:299] Negotiation successful
0808 02:24:28.158819 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260808 02:24:28.160175 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:28.140043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:41745 (local address 127.19.231.1:48245)
0808 02:24:28.141055 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:28.141110 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:28.143002 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:28.143786 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:28.143794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:28.144353 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:28.144961 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:28.144980 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.146468 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.146472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:28.147926 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:28.147940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.149017 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.149027 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:28.149400 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:28.150204 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:28.150228 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:28.152907 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:28.156136 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:28.156145 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:28.156161 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:28.156591 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:28.156981 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:28.156986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:28.156990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:28.157184 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:28.157757 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:28.157765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:28.158156 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 02:24:28.158417 (+ 261us) client_negotiation.cc:240] Negotiation successful
0808 02:24:28.158980 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":329,"threads_started":1}
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
I20260808 02:24:28.161873 23169 heartbeater.cc:347] Connected to a master server at 127.19.231.62:41745
I20260808 02:24:28.162201 23169 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:28.163053 23169 heartbeater.cc:511] Master 127.19.231.62:41745 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
I20260808 02:24:28.165551 22966 ts_manager.cc:194] Registered new tserver with Master: 498016ba9e864791bd0e14e7422de2e9 (127.19.231.1:45695)
I20260808 02:24:28.167856 22966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:48245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:28.192282 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41745
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:28.526899 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:28.527220 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:28.527405 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:28.534658 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:28.534808 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:28.534996 23175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:28.544667 23175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:28.545972 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:28.547379 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:28.561026 23181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:28.561041 23182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:28.561470 23184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:28.562208 23175 server_base.cc:1034] running on GCE node
I20260808 02:24:28.562944 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:28.563997 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:28.565223 23175 hybrid_clock.cc:648] HybridClock initialized: now 1786155868565174 us; error 67 us; skew 500 ppm
I20260808 02:24:28.565546 23175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:28 dist-test-slave-pgkr krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:28.568806 23175 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:28.570849 23175 webserver.cc:460] Webserver started at http://127.19.231.2:38369/ using document root <none> and password file <none>
I20260808 02:24:28.571527 23175 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:28.571660 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:28.572032 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:28.575295 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "b4b806d0ef7d4332aea80c1fe9c63b7f"
format_stamp: "Formatted at 2026-08-08 02:24:28 on dist-test-slave-pgkr"
server_key: "f8a6b10c002b931dc991713ff4ee2db5"
server_key_iv: "ce49d3b216a6971bb3ded2c5e72841a4"
server_key_version: "encryptionkey@0"
I20260808 02:24:28.576098 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "b4b806d0ef7d4332aea80c1fe9c63b7f"
format_stamp: "Formatted at 2026-08-08 02:24:28 on dist-test-slave-pgkr"
server_key: "f8a6b10c002b931dc991713ff4ee2db5"
server_key_iv: "ce49d3b216a6971bb3ded2c5e72841a4"
server_key_version: "encryptionkey@0"
I20260808 02:24:28.581795 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:24:28.586256 23191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:28.587801 23175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:28.587980 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "b4b806d0ef7d4332aea80c1fe9c63b7f"
format_stamp: "Formatted at 2026-08-08 02:24:28 on dist-test-slave-pgkr"
server_key: "f8a6b10c002b931dc991713ff4ee2db5"
server_key_iv: "ce49d3b216a6971bb3ded2c5e72841a4"
server_key_version: "encryptionkey@0"
I20260808 02:24:28.588204 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:28.611945 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:28.616760 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:28.617074 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:28.618093 23175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:28.619567 23175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:28.619645 23175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:28.619800 23175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:28.619851 23175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:28.657780 23175 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:44047
I20260808 02:24:28.657976 23313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:44047 every 8 connection(s)
I20260808 02:24:28.659221 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:28.665216 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23175
I20260808 02:24:28.665431 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:28.665869 20380 external_mini_cluster.cc:1428] Setting key d28c9b262a01b937e3bb5b15dec4079f
Aug 08 02:24:28 dist-test-slave-pgkr krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:28.682655 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:28.662328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:40571 (local address 127.19.231.62:41745)
0808 02:24:28.662842 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:28.662847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:28.663886 (+ 1039us) server_negotiation.cc:407] Connection header received
0808 02:24:28.665947 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:28.665955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:28.666048 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:28.666170 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:28.667877 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.668589 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.670124 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.670843 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.670999 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:28.675298 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:28.675303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:28.675307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:28.675339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:28.678911 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:28.679668 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:28.679677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:28.679684 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:28.679848 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:28.680388 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:28.680398 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:28.680405 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:28.680875 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:28.681056 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:28.681760 (+ 704us) server_negotiation.cc:299] Negotiation successful
0808 02:24:28.682080 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":256,"threads_started":1}
I20260808 02:24:28.683154 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:28.662964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:41745 (local address 127.19.231.2:40571)
0808 02:24:28.663720 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:28.663765 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:28.665689 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:28.666407 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:28.666415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:28.667072 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:28.667712 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:28.667739 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.668751 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.668757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:28.669971 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:28.669983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:28.671121 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:28.671129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:28.671374 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:28.672203 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:28.672228 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:28.675078 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:28.679079 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:28.679086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:28.679098 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:28.679516 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:28.679987 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:28.679990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:28.679993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:28.680205 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:28.681020 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:28.681026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:28.681341 (+ 315us) client_negotiation.cc:769] Sending connection context
0808 02:24:28.681577 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 02:24:28.681969 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":286,"threads_started":1}
I20260808 02:24:28.684775 23314 heartbeater.cc:347] Connected to a master server at 127.19.231.62:41745
I20260808 02:24:28.685144 23314 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:28.686141 23314 heartbeater.cc:511] Master 127.19.231.62:41745 requested a full tablet report, sending...
I20260808 02:24:28.687976 22966 ts_manager.cc:194] Registered new tserver with Master: b4b806d0ef7d4332aea80c1fe9c63b7f (127.19.231.2:44047)
I20260808 02:24:28.689199 22966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:40571
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:28.720257 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41745
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:29.059065 23320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:29.059396 23320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:29.059543 23320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:29.067013 23320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:29.067122 23320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:29.067236 23320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:29.077396 23320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:29.078681 23320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:29.080068 23320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:29.092991 23326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:29.093242 23327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:29.093559 23330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:29.094341 23320 server_base.cc:1034] running on GCE node
I20260808 02:24:29.095081 23320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:29.096045 23320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:29.097266 23320 hybrid_clock.cc:648] HybridClock initialized: now 1786155869097214 us; error 61 us; skew 500 ppm
I20260808 02:24:29.097581 23320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:29 dist-test-slave-pgkr krb5kdc[22919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155869, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:29.100977 23320 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:29.102514 23320 webserver.cc:460] Webserver started at http://127.19.231.3:33007/ using document root <none> and password file <none>
I20260808 02:24:29.103253 23320 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:29.103399 23320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:29.103820 23320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:29.107231 23320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "e7b8574d75ff438cb7431aac32875b0d"
format_stamp: "Formatted at 2026-08-08 02:24:29 on dist-test-slave-pgkr"
server_key: "de8e813874095fa9e48bbcbb781d1427"
server_key_iv: "3d6e4f0d9cb850a927c89612e6ad64e9"
server_key_version: "encryptionkey@0"
I20260808 02:24:29.107985 23320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "e7b8574d75ff438cb7431aac32875b0d"
format_stamp: "Formatted at 2026-08-08 02:24:29 on dist-test-slave-pgkr"
server_key: "de8e813874095fa9e48bbcbb781d1427"
server_key_iv: "3d6e4f0d9cb850a927c89612e6ad64e9"
server_key_version: "encryptionkey@0"
I20260808 02:24:29.113322 23320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:29.116814 23338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:29.118192 23320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:29.118366 23320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "e7b8574d75ff438cb7431aac32875b0d"
format_stamp: "Formatted at 2026-08-08 02:24:29 on dist-test-slave-pgkr"
server_key: "de8e813874095fa9e48bbcbb781d1427"
server_key_iv: "3d6e4f0d9cb850a927c89612e6ad64e9"
server_key_version: "encryptionkey@0"
I20260808 02:24:29.118568 23320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:29.136479 23320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:29.140641 23320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:29.140971 23320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:29.141897 23320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:29.143306 23320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:29.143411 23320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:29.143584 23320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:29.143668 23320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:29.171927 23169 heartbeater.cc:503] Master 127.19.231.62:41745 was elected leader, sending a full tablet report...
I20260808 02:24:29.179466 23320 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:42819
I20260808 02:24:29.179557 23456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:42819 every 8 connection(s)
I20260808 02:24:29.180770 23320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:29.181945 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23320
I20260808 02:24:29.182170 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:29.182709 20380 external_mini_cluster.cc:1428] Setting key f4a4ab125e237583cea1969152373e0d
Aug 08 02:24:29 dist-test-slave-pgkr krb5kdc[22919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155869, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:29.202492 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:29.183834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:45105 (local address 127.19.231.62:41745)
0808 02:24:29.184272 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:29.184278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:29.185733 (+ 1455us) server_negotiation.cc:407] Connection header received
0808 02:24:29.187915 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:29.187919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:29.187996 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:29.188085 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:29.189653 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:29.190410 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:29.192200 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:29.192591 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:29.192737 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:29.196435 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:29.196439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:29.196443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:29.196475 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:29.199649 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:29.200343 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:29.200347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:29.200349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:29.200518 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:29.200938 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:29.200941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:29.200943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:29.201215 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:29.201339 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:29.202035 (+ 696us) server_negotiation.cc:299] Negotiation successful
0808 02:24:29.202219 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":199,"threads_started":1}
I20260808 02:24:29.203548 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:29.184491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:41745 (local address 127.19.231.3:45105)
0808 02:24:29.185541 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:29.185606 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:29.187632 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:29.188312 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:29.188321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:29.188910 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:29.189495 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:29.189524 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:29.190664 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:29.190670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:29.192041 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:29.192054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:29.192851 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:29.192856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:29.193057 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:29.193726 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:29.193752 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:29.196221 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:29.199777 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:29.199784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:29.199797 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:29.200232 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:29.200638 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:29.200641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:29.200644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:29.200837 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:29.201328 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:29.201334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:29.201708 (+ 374us) client_negotiation.cc:769] Sending connection context
0808 02:24:29.201944 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 02:24:29.202342 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":396,"threads_started":1}
I20260808 02:24:29.204999 23457 heartbeater.cc:347] Connected to a master server at 127.19.231.62:41745
I20260808 02:24:29.205305 23457 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:29.206059 23457 heartbeater.cc:511] Master 127.19.231.62:41745 requested a full tablet report, sending...
I20260808 02:24:29.207595 22966 ts_manager.cc:194] Registered new tserver with Master: e7b8574d75ff438cb7431aac32875b0d (127.19.231.3:42819)
I20260808 02:24:29.208428 22966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:45105
I20260808 02:24:29.211616 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:24:29.249106 23459 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:29.245907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.19.231.62:41745)
0808 02:24:29.246179 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:29.246186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:29.246777 (+ 591us) server_negotiation.cc:407] Connection header received
0808 02:24:29.246987 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:29.246993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:29.247076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:29.247189 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:29.248824 (+ 1635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38678: BlockingRecv error: recv got EOF from 127.0.0.1:38678 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:24:29.253324 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23030
W20260808 02:24:29.256599 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23030: Not authorized: Client connection negotiation failed: client connection to 127.19.231.1:45695: server requires authentication, but client does not have Kerberos credentials available
I20260808 02:24:29.256780 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23030
I20260808 02:24:29.277606 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23175
W20260808 02:24:29.280359 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23175: Not authorized: Client connection negotiation failed: client connection to 127.19.231.2:44047: server requires authentication, but client does not have Kerberos credentials available
I20260808 02:24:29.280534 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23175
I20260808 02:24:29.300101 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23320
W20260808 02:24:29.303113 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23320: Not authorized: Client connection negotiation failed: client connection to 127.19.231.3:42819: server requires authentication, but client does not have Kerberos credentials available
I20260808 02:24:29.303282 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23320
I20260808 02:24:29.321995 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 22933
I20260808 02:24:29.323369 22998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:29.429477 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 22933
2026-08-08T02:24:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3726 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:29 dist-test-slave-pgkr krb5kdc[23476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:29 dist-test-slave-pgkr krb5kdc[23476](info): set up 2 sockets
Aug 08 02:24:29 dist-test-slave-pgkr krb5kdc[23476](info): commencing operation
krb5kdc: starting...
W20260808 02:24:30.562145 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:30 dist-test-slave-pgkr krb5kdc[23476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:30.794552 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:39245
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:31.142132 23489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:31.142441 23489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:31.142580 23489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:31.149937 23489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:31.150039 23489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:31.150085 23489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:31.150161 23489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:31.150251 23489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:31.160280 23489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:31.161633 23489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:31.163062 23489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:31.175354 23496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:31.175324 23497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:31.175405 23499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:31.176452 23489 server_base.cc:1034] running on GCE node
I20260808 02:24:31.177198 23489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:31.178599 23489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:31.179876 23489 hybrid_clock.cc:648] HybridClock initialized: now 1786155871179816 us; error 53 us; skew 500 ppm
I20260808 02:24:31.180263 23489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:31 dist-test-slave-pgkr krb5kdc[23476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:31.187803 23489 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:31.189498 23489 webserver.cc:460] Webserver started at http://127.19.231.62:42063/ using document root <none> and password file <none>
I20260808 02:24:31.190245 23489 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:31.190400 23489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:31.190819 23489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:31.194041 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "7220b9b9498f47299aa1b07241930893"
format_stamp: "Formatted at 2026-08-08 02:24:31 on dist-test-slave-pgkr"
server_key: "efc8071e0c0ef06f3676400585234e1d"
server_key_iv: "0317dc1247d6233167761be83a7ef90e"
server_key_version: "encryptionkey@0"
I20260808 02:24:31.194955 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "7220b9b9498f47299aa1b07241930893"
format_stamp: "Formatted at 2026-08-08 02:24:31 on dist-test-slave-pgkr"
server_key: "efc8071e0c0ef06f3676400585234e1d"
server_key_iv: "0317dc1247d6233167761be83a7ef90e"
server_key_version: "encryptionkey@0"
I20260808 02:24:31.200296 23489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:24:31.204017 23508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:31.205410 23489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:31.205601 23489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "7220b9b9498f47299aa1b07241930893"
format_stamp: "Formatted at 2026-08-08 02:24:31 on dist-test-slave-pgkr"
server_key: "efc8071e0c0ef06f3676400585234e1d"
server_key_iv: "0317dc1247d6233167761be83a7ef90e"
server_key_version: "encryptionkey@0"
I20260808 02:24:31.205814 23489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:31.231374 23489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:31.235584 23489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:31.235891 23489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:31.258657 23489 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:39245
I20260808 02:24:31.258697 23560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:39245 every 8 connection(s)
I20260808 02:24:31.260110 23489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:31.264089 23561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:31.268005 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23489
I20260808 02:24:31.268309 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:31.269414 20380 external_mini_cluster.cc:1428] Setting key c5e22d342624da451c5c6a2faf096437
I20260808 02:24:31.278501 23561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893: Bootstrap starting.
I20260808 02:24:31.282548 23561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:31.283972 23561 log.cc:824] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:24:31 dist-test-slave-pgkr krb5kdc[23476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:31.286993 23561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893: No bootstrap required, opened a new log
I20260808 02:24:31.293061 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:31.271553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.19.231.62:39245)
0808 02:24:31.272385 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:31.272398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:31.272442 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:24:31.274148 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:31.274180 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:31.274665 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:31.275033 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:31.276933 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:31.278225 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:31.279993 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:31.280667 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:31.280816 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:31.285776 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:31.285784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:31.285799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:31.285833 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:31.288936 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:31.289657 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:31.289664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:31.289669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:31.289840 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:31.290232 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:31.290236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:31.290238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:31.290831 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:31.291172 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:31.291588 (+ 416us) server_negotiation.cc:299] Negotiation successful
0808 02:24:31.292001 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":367,"threads_started":1}
I20260808 02:24:31.295701 23561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220b9b9498f47299aa1b07241930893" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39245 } }
I20260808 02:24:31.296134 23561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:31.296245 23561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7220b9b9498f47299aa1b07241930893, State: Initialized, Role: FOLLOWER
I20260808 02:24:31.296846 23561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220b9b9498f47299aa1b07241930893" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39245 } }
I20260808 02:24:31.297096 23561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:31.297246 23561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:31.297308 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:24:31.297448 23561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:31.299716 23561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220b9b9498f47299aa1b07241930893" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39245 } }
I20260808 02:24:31.300220 23561 leader_election.cc:302] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7220b9b9498f47299aa1b07241930893; no voters:
I20260808 02:24:31.300906 23561 leader_election.cc:288] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:31.301028 23566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:31.302901 23566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [term 1 LEADER]: Becoming Leader. State: Replica: 7220b9b9498f47299aa1b07241930893, State: Running, Role: LEADER
I20260808 02:24:31.303573 23561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:31.303579 23566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220b9b9498f47299aa1b07241930893" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39245 } }
I20260808 02:24:31.308188 23569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7220b9b9498f47299aa1b07241930893. Latest consensus state: current_term: 1 leader_uuid: "7220b9b9498f47299aa1b07241930893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220b9b9498f47299aa1b07241930893" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39245 } } }
I20260808 02:24:31.308641 23569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:31.309336 23567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7220b9b9498f47299aa1b07241930893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220b9b9498f47299aa1b07241930893" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39245 } } }
I20260808 02:24:31.309696 23567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:31.311739 23578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:31.317610 23578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:31.330256 23578 catalog_manager.cc:1347] Generated new cluster ID: e1e5985731b64ec3b5486838379ef7d2
I20260808 02:24:31.330380 23578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:31.360133 23578 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:31.362188 23578 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:31.369560 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:31.360319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.19.231.62:39245)
0808 02:24:31.360506 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:31.360515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:31.361157 (+ 642us) server_negotiation.cc:407] Connection header received
0808 02:24:31.361383 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:31.361388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:31.362190 (+ 802us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:31.362311 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:31.363480 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:31.364236 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:31.365524 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:31.365964 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:31.366102 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:31.367279 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:31.367284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:31.367288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:31.367931 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:31.367992 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:31.368023 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:31.368147 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:31.368294 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 02:24:31.368388 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:24:31.371402 23578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7220b9b9498f47299aa1b07241930893: Generated new TSK 0
I20260808 02:24:31.372450 23578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:24:31.383463 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23489
I20260808 02:24:31.384598 23555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:31.477952 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23489
2026-08-08T02:24:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2043 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:31 dist-test-slave-pgkr krb5kdc[23598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:31 dist-test-slave-pgkr krb5kdc[23598](info): set up 2 sockets
Aug 08 02:24:31 dist-test-slave-pgkr krb5kdc[23598](info): commencing operation
krb5kdc: starting...
W20260808 02:24:32.618430 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:32 dist-test-slave-pgkr krb5kdc[23598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:32.854950 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44753
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:33.205152 23611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:33.205437 23611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:33.205595 23611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:33.213066 23611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:33.213173 23611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:33.213254 23611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:33.213351 23611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:33.213436 23611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:33.223472 23611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:33.224874 23611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:33.226224 23611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:33.239033 23616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:33.238988 23619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:33.239063 23617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:33.239665 23611 server_base.cc:1034] running on GCE node
I20260808 02:24:33.240579 23611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:33.242087 23611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:33.243340 23611 hybrid_clock.cc:648] HybridClock initialized: now 1786155873243280 us; error 38 us; skew 500 ppm
I20260808 02:24:33.243757 23611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:33 dist-test-slave-pgkr krb5kdc[23598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155873, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:33.248027 23611 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:33.250670 23611 webserver.cc:460] Webserver started at http://127.19.231.62:44517/ using document root <none> and password file <none>
I20260808 02:24:33.251713 23611 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:33.251864 23611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:33.252279 23611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:33.256210 23611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "81235f08a15e496fa6adb74c074a2035"
format_stamp: "Formatted at 2026-08-08 02:24:33 on dist-test-slave-pgkr"
server_key: "f32edfe5cbdf334f786baf8dbfdccbb3"
server_key_iv: "66dcc545eeeb42dd1a6489432aec8dce"
server_key_version: "encryptionkey@0"
I20260808 02:24:33.256989 23611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "81235f08a15e496fa6adb74c074a2035"
format_stamp: "Formatted at 2026-08-08 02:24:33 on dist-test-slave-pgkr"
server_key: "f32edfe5cbdf334f786baf8dbfdccbb3"
server_key_iv: "66dcc545eeeb42dd1a6489432aec8dce"
server_key_version: "encryptionkey@0"
I20260808 02:24:33.262231 23611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:24:33.265717 23626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:33.267139 23611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260808 02:24:33.267334 23611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "81235f08a15e496fa6adb74c074a2035"
format_stamp: "Formatted at 2026-08-08 02:24:33 on dist-test-slave-pgkr"
server_key: "f32edfe5cbdf334f786baf8dbfdccbb3"
server_key_iv: "66dcc545eeeb42dd1a6489432aec8dce"
server_key_version: "encryptionkey@0"
I20260808 02:24:33.267532 23611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:33.281258 23611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:33.285082 23611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:33.285375 23611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:33.304425 23611 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:44753
I20260808 02:24:33.304447 23680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:44753 every 8 connection(s)
I20260808 02:24:33.305905 23611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:33.308450 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23611
I20260808 02:24:33.308746 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:33.309530 20380 external_mini_cluster.cc:1428] Setting key d904f5cfe1f51965524185a795f6e199
I20260808 02:24:33.310047 23681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:33.324630 23681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035: Bootstrap starting.
Aug 08 02:24:33 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:33.328567 23681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:33.329929 23681 log.cc:824] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:33.333630 23681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035: No bootstrap required, opened a new log
I20260808 02:24:33.335266 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:33.311595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.19.231.62:44753)
0808 02:24:33.312431 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:33.312443 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:33.312485 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:24:33.314805 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:33.314830 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:33.315226 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:33.315545 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:33.317368 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:33.318675 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:33.320516 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:33.321293 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:33.321498 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:33.326251 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:33.326262 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:33.326280 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:33.326342 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:33.330303 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:33.331414 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:33.331420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:33.331426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:33.331643 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:33.332069 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:33.332073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:33.332076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:33.332769 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:33.333063 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:33.333458 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 02:24:33.333890 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":383,"threads_started":1}
I20260808 02:24:33.342010 23681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81235f08a15e496fa6adb74c074a2035" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44753 } }
I20260808 02:24:33.342346 23681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:33.342417 23681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81235f08a15e496fa6adb74c074a2035, State: Initialized, Role: FOLLOWER
I20260808 02:24:33.343008 23681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81235f08a15e496fa6adb74c074a2035" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44753 } }
I20260808 02:24:33.343283 23681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:33.343444 23681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:33.343621 23681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:33.345803 23681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81235f08a15e496fa6adb74c074a2035" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44753 } }
I20260808 02:24:33.346280 23681 leader_election.cc:302] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81235f08a15e496fa6adb74c074a2035; no voters:
I20260808 02:24:33.346904 23681 leader_election.cc:288] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:33.347035 23686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:33.348215 23686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [term 1 LEADER]: Becoming Leader. State: Replica: 81235f08a15e496fa6adb74c074a2035, State: Running, Role: LEADER
I20260808 02:24:33.348846 23686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81235f08a15e496fa6adb74c074a2035" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44753 } }
I20260808 02:24:33.349169 23681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:33.351701 23687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81235f08a15e496fa6adb74c074a2035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81235f08a15e496fa6adb74c074a2035" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44753 } } }
I20260808 02:24:33.352100 23687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:33.352312 23688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81235f08a15e496fa6adb74c074a2035. Latest consensus state: current_term: 1 leader_uuid: "81235f08a15e496fa6adb74c074a2035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81235f08a15e496fa6adb74c074a2035" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44753 } } }
I20260808 02:24:33.352646 23688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:33.356142 23694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:33.360867 23694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:33.369366 23694 catalog_manager.cc:1347] Generated new cluster ID: 335351912c2a415481cc43811ea58727
I20260808 02:24:33.369477 23694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:33.380954 23694 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:33.382771 23694 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:33.397771 23694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81235f08a15e496fa6adb74c074a2035: Generated new TSK 0
I20260808 02:24:33.398753 23694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:33.459924 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:44753
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:33.806862 23707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:33.807128 23707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:33.807579 23707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:33.815001 23707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:33.815099 23707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:33.815304 23707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:33.825028 23707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:44753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:33.826341 23707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:33.827713 23707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:33.842391 23712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:33.842356 23715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:33.842442 23713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:33.843474 23707 server_base.cc:1034] running on GCE node
I20260808 02:24:33.844112 23707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:33.845019 23707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:33.846215 23707 hybrid_clock.cc:648] HybridClock initialized: now 1786155873846174 us; error 29 us; skew 500 ppm
I20260808 02:24:33.846531 23707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:33 dist-test-slave-pgkr krb5kdc[23598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155873, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:33.849892 23707 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:33.851500 23707 webserver.cc:460] Webserver started at http://127.19.231.1:35713/ using document root <none> and password file <none>
I20260808 02:24:33.852205 23707 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:33.852373 23707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:33.852687 23707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:33.855942 23707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "2e7c3fe0e01e4d2da448ac02f4b4dffd"
format_stamp: "Formatted at 2026-08-08 02:24:33 on dist-test-slave-pgkr"
server_key: "c763993309e212b74f188925f9dfe0ac"
server_key_iv: "5d93441816dfd3bbc34bcd6a7d5bc409"
server_key_version: "encryptionkey@0"
I20260808 02:24:33.856703 23707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "2e7c3fe0e01e4d2da448ac02f4b4dffd"
format_stamp: "Formatted at 2026-08-08 02:24:33 on dist-test-slave-pgkr"
server_key: "c763993309e212b74f188925f9dfe0ac"
server_key_iv: "5d93441816dfd3bbc34bcd6a7d5bc409"
server_key_version: "encryptionkey@0"
I20260808 02:24:33.861728 23707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 02:24:33.865155 23722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:33.866501 23707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 02:24:33.866719 23707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "2e7c3fe0e01e4d2da448ac02f4b4dffd"
format_stamp: "Formatted at 2026-08-08 02:24:33 on dist-test-slave-pgkr"
server_key: "c763993309e212b74f188925f9dfe0ac"
server_key_iv: "5d93441816dfd3bbc34bcd6a7d5bc409"
server_key_version: "encryptionkey@0"
I20260808 02:24:33.866900 23707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:33.886065 23707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:33.890082 23707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:33.890416 23707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:33.891413 23707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:33.892822 23707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:33.892895 23707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:33.893038 23707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:33.893123 23707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:33.927192 23707 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:39951
I20260808 02:24:33.927227 23842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:39951 every 8 connection(s)
I20260808 02:24:33.928488 23707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:33.933405 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23707
I20260808 02:24:33.933637 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:33.934274 20380 external_mini_cluster.cc:1428] Setting key ed49b31923c8389d6532a30fd3f5ca86
Aug 08 02:24:33 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155873, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:33.949698 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:33.931058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:39465 (local address 127.19.231.62:44753)
0808 02:24:33.931520 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:33.931525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:33.932557 (+ 1032us) server_negotiation.cc:407] Connection header received
0808 02:24:33.934525 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:33.934531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:33.934669 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:33.934796 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:33.936449 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:33.937519 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:33.939108 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:33.939804 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:33.939963 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:33.943521 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:33.943530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:33.943538 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:33.943593 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:33.946360 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:33.947309 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:33.947313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:33.947315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:33.947436 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:33.947923 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:33.947926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:33.947928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:33.948169 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:33.948275 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:33.949176 (+ 901us) server_negotiation.cc:299] Negotiation successful
0808 02:24:33.949425 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":226,"threads_started":1}
I20260808 02:24:33.951263 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:33.931653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:44753 (local address 127.19.231.1:39465)
0808 02:24:33.932367 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:33.932407 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:33.934230 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:33.935031 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:33.935042 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:33.935598 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:33.936270 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:33.936305 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:33.937675 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:33.937679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:33.938962 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:33.938974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:33.940025 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:33.940029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:33.940231 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:33.940920 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:33.940947 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:33.943299 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:33.946502 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:33.946514 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:33.946533 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:33.947198 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:33.947547 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:33.947554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:33.947560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:33.947808 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:33.948299 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:33.948310 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:33.948771 (+ 461us) client_negotiation.cc:769] Sending connection context
0808 02:24:33.949075 (+ 304us) client_negotiation.cc:240] Negotiation successful
0808 02:24:33.949672 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":265,"threads_started":1}
I20260808 02:24:33.952895 23843 heartbeater.cc:347] Connected to a master server at 127.19.231.62:44753
I20260808 02:24:33.953218 23843 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:33.953965 23843 heartbeater.cc:511] Master 127.19.231.62:44753 requested a full tablet report, sending...
I20260808 02:24:33.956338 23643 ts_manager.cc:194] Registered new tserver with Master: 2e7c3fe0e01e4d2da448ac02f4b4dffd (127.19.231.1:39951)
I20260808 02:24:33.957796 23643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:39465
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:33.987974 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:44753
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:34.321288 23849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:34.321615 23849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:34.321800 23849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:34.329416 23849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:34.329567 23849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:34.329748 23849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:34.339687 23849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:44753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:34.341049 23849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:34.342443 23849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:34.356181 23854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:34.356317 23857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:34.356181 23855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:34.357963 23849 server_base.cc:1034] running on GCE node
I20260808 02:24:34.358598 23849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:34.359597 23849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:34.360818 23849 hybrid_clock.cc:648] HybridClock initialized: now 1786155874360762 us; error 49 us; skew 500 ppm
I20260808 02:24:34.361158 23849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:34 dist-test-slave-pgkr krb5kdc[23598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155874, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:34.364486 23849 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:34.366030 23849 webserver.cc:460] Webserver started at http://127.19.231.2:37119/ using document root <none> and password file <none>
I20260808 02:24:34.366799 23849 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:34.366947 23849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:34.367281 23849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:34.370276 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "e4493eaeeb604d00915248d7ac471f9a"
format_stamp: "Formatted at 2026-08-08 02:24:34 on dist-test-slave-pgkr"
server_key: "4d73fda90c86502bfc9c89b492ea9900"
server_key_iv: "42e99614a67b5c0525accafbe49b8b15"
server_key_version: "encryptionkey@0"
I20260808 02:24:34.371147 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "e4493eaeeb604d00915248d7ac471f9a"
format_stamp: "Formatted at 2026-08-08 02:24:34 on dist-test-slave-pgkr"
server_key: "4d73fda90c86502bfc9c89b492ea9900"
server_key_iv: "42e99614a67b5c0525accafbe49b8b15"
server_key_version: "encryptionkey@0"
I20260808 02:24:34.376482 23849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:24:34.379808 23865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:34.381111 23849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:34.381295 23849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "e4493eaeeb604d00915248d7ac471f9a"
format_stamp: "Formatted at 2026-08-08 02:24:34 on dist-test-slave-pgkr"
server_key: "4d73fda90c86502bfc9c89b492ea9900"
server_key_iv: "42e99614a67b5c0525accafbe49b8b15"
server_key_version: "encryptionkey@0"
I20260808 02:24:34.381484 23849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:34.409252 23849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:34.413753 23849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:34.414059 23849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:34.415072 23849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:34.416425 23849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:34.416530 23849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:34.416684 23849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:34.416757 23849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:34.450665 23849 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:44201
I20260808 02:24:34.450754 23983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:44201 every 8 connection(s)
I20260808 02:24:34.452024 23849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:34.460636 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23849
I20260808 02:24:34.460817 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:34.461318 20380 external_mini_cluster.cc:1428] Setting key 6759d78326ac7a01d6b6a39eb8c0b32a
Aug 08 02:24:34 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155874, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:34.472455 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:34.454742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:41507 (local address 127.19.231.62:44753)
0808 02:24:34.455199 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:34.455203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:34.456252 (+ 1049us) server_negotiation.cc:407] Connection header received
0808 02:24:34.458100 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:34.458105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:34.458193 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:34.458311 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:34.459829 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.460549 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.462304 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.462953 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.463084 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:34.466483 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:34.466487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:34.466491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:34.466521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:34.469132 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:34.469859 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:34.469870 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:34.469873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:34.469996 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:34.470440 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:34.470448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:34.470451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:34.470831 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:34.470994 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:34.471790 (+ 796us) server_negotiation.cc:299] Negotiation successful
0808 02:24:34.472016 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":255,"threads_started":1}
I20260808 02:24:34.473336 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:34.455350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:44753 (local address 127.19.231.2:41507)
0808 02:24:34.456092 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:34.456131 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:34.457827 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:34.458541 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:34.458549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:34.459102 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:34.459661 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:34.459678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.460700 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.460707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:34.462116 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:34.462138 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.463189 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.463196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:34.463412 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:34.464073 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:34.464094 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:34.466240 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:34.469290 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:34.469298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:34.469314 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:34.469731 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:34.470108 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:34.470113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:34.470117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:34.470319 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:34.470963 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:34.470971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:34.471355 (+ 384us) client_negotiation.cc:769] Sending connection context
0808 02:24:34.471620 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 02:24:34.472185 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":270,"threads_started":1}
I20260808 02:24:34.474725 23984 heartbeater.cc:347] Connected to a master server at 127.19.231.62:44753
I20260808 02:24:34.475049 23984 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:34.475901 23984 heartbeater.cc:511] Master 127.19.231.62:44753 requested a full tablet report, sending...
I20260808 02:24:34.477450 23643 ts_manager.cc:194] Registered new tserver with Master: e4493eaeeb604d00915248d7ac471f9a (127.19.231.2:44201)
I20260808 02:24:34.478258 23643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:41507
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:34.514089 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:44753
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:34.857877 23990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:34.858166 23990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:34.858270 23990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:34.865557 23990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:34.865679 23990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:34.865795 23990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:34.876211 23990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:44753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:34.877568 23990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:34.878950 23990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:34.893150 23995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:34.893239 23996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:34.893193 23998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:34.896142 23990 server_base.cc:1034] running on GCE node
I20260808 02:24:34.896824 23990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:34.897859 23990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:34.899082 23990 hybrid_clock.cc:648] HybridClock initialized: now 1786155874899033 us; error 30 us; skew 500 ppm
I20260808 02:24:34.899433 23990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:34 dist-test-slave-pgkr krb5kdc[23598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155874, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:34.902733 23990 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:34.904345 23990 webserver.cc:460] Webserver started at http://127.19.231.3:45997/ using document root <none> and password file <none>
I20260808 02:24:34.905094 23990 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:34.905241 23990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:34.905555 23990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:34.908572 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "e8f5b1c21a29449cb9c85ad84c3857e9"
format_stamp: "Formatted at 2026-08-08 02:24:34 on dist-test-slave-pgkr"
server_key: "53573ae054fdea2ae3748ce031645ee9"
server_key_iv: "add0e2bb4278f61954088500755f5892"
server_key_version: "encryptionkey@0"
I20260808 02:24:34.909389 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f5b1c21a29449cb9c85ad84c3857e9"
format_stamp: "Formatted at 2026-08-08 02:24:34 on dist-test-slave-pgkr"
server_key: "53573ae054fdea2ae3748ce031645ee9"
server_key_iv: "add0e2bb4278f61954088500755f5892"
server_key_version: "encryptionkey@0"
I20260808 02:24:34.914873 23990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 02:24:34.918350 24005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:34.919792 23990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:34.919975 23990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "e8f5b1c21a29449cb9c85ad84c3857e9"
format_stamp: "Formatted at 2026-08-08 02:24:34 on dist-test-slave-pgkr"
server_key: "53573ae054fdea2ae3748ce031645ee9"
server_key_iv: "add0e2bb4278f61954088500755f5892"
server_key_version: "encryptionkey@0"
I20260808 02:24:34.920182 23990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:34.932717 23990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:34.936836 23990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:34.937173 23990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:34.938184 23990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:34.939760 23990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:34.939881 23990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:34.940044 23990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:34.940124 23990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:34.960839 23843 heartbeater.cc:503] Master 127.19.231.62:44753 was elected leader, sending a full tablet report...
I20260808 02:24:34.977725 23990 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:46207
I20260808 02:24:34.977804 24127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:46207 every 8 connection(s)
I20260808 02:24:34.979262 23990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:34.985001 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 23990
I20260808 02:24:34.985277 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:34.985977 20380 external_mini_cluster.cc:1428] Setting key 797d10ca7ed7c000c95ea6ca1b4e74c3
Aug 08 02:24:34 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155874, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:34.999454 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:34.981911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:49547 (local address 127.19.231.62:44753)
0808 02:24:34.982423 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:34.982430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:34.983536 (+ 1106us) server_negotiation.cc:407] Connection header received
0808 02:24:34.985426 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:34.985433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:34.985549 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:34.985688 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:34.987389 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.988337 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.989996 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.990388 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.990523 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:34.993890 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:34.993894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:34.993898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:34.993929 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:34.996429 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:34.997134 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:34.997144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:34.997147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:34.997268 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:34.997694 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:34.997699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:34.997708 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:34.997962 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:34.998127 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:34.998856 (+ 729us) server_negotiation.cc:299] Negotiation successful
0808 02:24:34.999076 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":233,"threads_started":1}
I20260808 02:24:35.000595 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:34.982578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:44753 (local address 127.19.231.3:49547)
0808 02:24:34.983360 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:34.983399 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:34.985098 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:34.985899 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:34.985911 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:34.986492 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:34.987220 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:34.987238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.988490 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.988495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:34.989847 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:34.989860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:34.990638 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:34.990646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:34.990873 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:34.991510 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:34.991532 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:34.993684 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:34.996574 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:34.996582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:34.996596 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:34.997003 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:34.997378 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:34.997383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:34.997387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:34.997568 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:34.998097 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:34.998105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:34.998430 (+ 325us) client_negotiation.cc:769] Sending connection context
0808 02:24:34.998738 (+ 308us) client_negotiation.cc:240] Negotiation successful
0808 02:24:34.999271 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":275,"threads_started":1}
I20260808 02:24:35.002161 24128 heartbeater.cc:347] Connected to a master server at 127.19.231.62:44753
I20260808 02:24:35.002429 24128 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:35.003134 24128 heartbeater.cc:511] Master 127.19.231.62:44753 requested a full tablet report, sending...
I20260808 02:24:35.004626 23642 ts_manager.cc:194] Registered new tserver with Master: e8f5b1c21a29449cb9c85ad84c3857e9 (127.19.231.3:46207)
I20260808 02:24:35.005656 23642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:49547
I20260808 02:24:35.014222 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[23598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:35.071128 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:35.055615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.19.231.62:44753)
0808 02:24:35.055882 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:35.055889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:35.056265 (+ 376us) server_negotiation.cc:407] Connection header received
0808 02:24:35.056450 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:35.056455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:35.056527 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:35.056616 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:35.058196 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.059015 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.060368 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.060762 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.060851 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:35.065293 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:35.065297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:35.065301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:35.065335 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:35.068041 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.068727 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.068730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.068733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.068851 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.069260 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.069263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.069264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.069717 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:35.070018 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:35.070443 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 02:24:35.070732 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260808 02:24:35.071843 23642 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50626
I20260808 02:24:35.073578 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23707
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:35.090102 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:35.074813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.19.231.1:39951)
0808 02:24:35.075605 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:35.075613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:35.075638 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:24:35.075697 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:35.075704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:35.075921 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:35.076056 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:35.077192 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.078250 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.079492 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.080189 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.080306 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:35.084350 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:35.084362 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:35.084377 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:35.084410 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:35.087579 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.088294 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.088301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.088303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.088428 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.088807 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.088810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.088812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.089136 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:35.089367 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:35.089620 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 02:24:35.089823 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":417,"threads_started":1}
W20260808 02:24:35.091439 23838 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36084
W20260808 02:24:35.092658 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23707: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:35.092888 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23707
I20260808 02:24:35.113143 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23849
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:35.130280 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:35.114176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.19.231.2:44201)
0808 02:24:35.114898 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:35.114906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:35.114934 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:24:35.115000 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:35.115006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:35.115241 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:35.115378 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:35.116628 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.117780 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.119133 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.119795 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.119946 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:35.124297 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:35.124310 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:35.124325 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:35.124364 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:35.127668 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.128271 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.128279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.128282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.128412 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.128748 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.128751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.128753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.129135 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:35.129396 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:35.129604 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 02:24:35.129821 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":346,"threads_started":1}
W20260808 02:24:35.131731 23979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49078
W20260808 02:24:35.132894 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23849: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:35.133068 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23849
I20260808 02:24:35.152812 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23990
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[23598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155875, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:35.170598 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:35.153955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.19.231.3:46207)
0808 02:24:35.154785 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:35.154792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:35.154821 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:24:35.154880 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:35.154886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:35.155096 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:35.155227 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:35.156896 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.157978 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.159402 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:35.159980 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:35.160076 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:35.164700 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:35.164712 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:35.164728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:35.164765 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:35.167852 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.168598 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.168605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.168608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.168747 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:35.169118 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:35.169121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:35.169123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:35.169463 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:35.169714 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:35.169996 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 02:24:35.170204 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":408,"threads_started":1}
W20260808 02:24:35.172197 24123 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56894
W20260808 02:24:35.173439 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:24:35.173597 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23990
I20260808 02:24:35.193322 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 23611
I20260808 02:24:35.194736 23676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:35.308267 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 23611
2026-08-08T02:24:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3835 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[24147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[24147](info): set up 2 sockets
Aug 08 02:24:35 dist-test-slave-pgkr krb5kdc[24147](info): commencing operation
krb5kdc: starting...
W20260808 02:24:36.450498 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:36 dist-test-slave-pgkr krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:36.684242 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:36507
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:37.025388 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:37.025688 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:37.025911 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:37.033252 24160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:37.033365 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:37.033478 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:37.033578 24160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:37.033663 24160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:37.043541 24160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:36507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:37.044953 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:37.046387 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:37.058552 24169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:37.058677 24167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:37.058602 24166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:37.059726 24160 server_base.cc:1034] running on GCE node
I20260808 02:24:37.060498 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:37.061939 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:37.063149 24160 hybrid_clock.cc:648] HybridClock initialized: now 1786155877063110 us; error 33 us; skew 500 ppm
I20260808 02:24:37.063519 24160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:37 dist-test-slave-pgkr krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155877, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:37.071247 24160 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:37.072921 24160 webserver.cc:460] Webserver started at http://127.19.231.62:41119/ using document root <none> and password file <none>
I20260808 02:24:37.073704 24160 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:37.073853 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:37.074194 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:37.077390 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "ba653cf156394f0386fd8b4f2bf09969"
format_stamp: "Formatted at 2026-08-08 02:24:37 on dist-test-slave-pgkr"
server_key: "aa3c866ee9c7bfa11cce6f13e077f2e1"
server_key_iv: "744fb0c266f093cacabf5ff2d8b929a7"
server_key_version: "encryptionkey@0"
I20260808 02:24:37.078189 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "ba653cf156394f0386fd8b4f2bf09969"
format_stamp: "Formatted at 2026-08-08 02:24:37 on dist-test-slave-pgkr"
server_key: "aa3c866ee9c7bfa11cce6f13e077f2e1"
server_key_iv: "744fb0c266f093cacabf5ff2d8b929a7"
server_key_version: "encryptionkey@0"
I20260808 02:24:37.083415 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:24:37.086934 24178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:37.088342 24160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:24:37.088546 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "ba653cf156394f0386fd8b4f2bf09969"
format_stamp: "Formatted at 2026-08-08 02:24:37 on dist-test-slave-pgkr"
server_key: "aa3c866ee9c7bfa11cce6f13e077f2e1"
server_key_iv: "744fb0c266f093cacabf5ff2d8b929a7"
server_key_version: "encryptionkey@0"
I20260808 02:24:37.088758 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:37.109789 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:37.113976 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:37.114300 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:37.133682 24160 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:36507
I20260808 02:24:37.133613 24233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:36507 every 8 connection(s)
I20260808 02:24:37.135344 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:37.136993 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 24160
I20260808 02:24:37.137239 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:37.138095 20380 external_mini_cluster.cc:1428] Setting key 8016ac44c3ed958b36e44539ca5dd8cb
I20260808 02:24:37.140363 24234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:37.154528 24234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969: Bootstrap starting.
Aug 08 02:24:37 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:37.157831 24234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:37.158942 24234 log.cc:824] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:37.162065 24234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969: No bootstrap required, opened a new log
I20260808 02:24:37.165661 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:37.140454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.19.231.62:36507)
0808 02:24:37.141448 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:37.141463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:37.141511 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:24:37.143627 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:37.143652 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:37.144121 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:37.144498 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:37.146415 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:37.148130 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:37.149899 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:37.150602 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:37.150842 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:37.156023 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:37.156037 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:37.156059 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:37.156106 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:37.160636 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:37.161553 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:37.161561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:37.161568 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:37.161766 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:37.162281 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:37.162286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:37.162290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:37.163046 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:37.163340 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:37.163768 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:24:37.164236 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":475,"threads_started":1}
I20260808 02:24:37.171001 24234 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba653cf156394f0386fd8b4f2bf09969" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 36507 } }
I20260808 02:24:37.171339 24234 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:37.171411 24234 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba653cf156394f0386fd8b4f2bf09969, State: Initialized, Role: FOLLOWER
I20260808 02:24:37.171888 24234 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba653cf156394f0386fd8b4f2bf09969" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 36507 } }
I20260808 02:24:37.172123 24234 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:37.172333 24234 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:37.172585 24234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:37.174808 24234 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba653cf156394f0386fd8b4f2bf09969" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 36507 } }
I20260808 02:24:37.175302 24234 leader_election.cc:302] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba653cf156394f0386fd8b4f2bf09969; no voters:
I20260808 02:24:37.175940 24234 leader_election.cc:288] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:37.176082 24239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:37.176863 24239 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [term 1 LEADER]: Becoming Leader. State: Replica: ba653cf156394f0386fd8b4f2bf09969, State: Running, Role: LEADER
I20260808 02:24:37.177371 24239 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba653cf156394f0386fd8b4f2bf09969" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 36507 } }
I20260808 02:24:37.178057 24234 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:37.180068 24240 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba653cf156394f0386fd8b4f2bf09969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba653cf156394f0386fd8b4f2bf09969" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 36507 } } }
I20260808 02:24:37.180462 24240 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:37.180542 24241 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba653cf156394f0386fd8b4f2bf09969. Latest consensus state: current_term: 1 leader_uuid: "ba653cf156394f0386fd8b4f2bf09969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba653cf156394f0386fd8b4f2bf09969" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 36507 } } }
I20260808 02:24:37.180905 24241 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:37.188911 24244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:37.194469 24244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:37.203640 24244 catalog_manager.cc:1347] Generated new cluster ID: 599c4038f42d4ffbac035e81df4e70a7
I20260808 02:24:37.203763 24244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:37.212267 24244 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:37.214176 24244 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:37.233179 24244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba653cf156394f0386fd8b4f2bf09969: Generated new TSK 0
I20260808 02:24:37.234234 24244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:37.294061 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:36507
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:37.631234 24260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:37.631505 24260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:37.631660 24260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:37.638989 24260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:37.639096 24260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:37.639305 24260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:37.649085 24260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:36507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:37.650436 24260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:37.651870 24260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:37.665627 24266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:37.665807 24268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:37.666015 24265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:37.666963 24260 server_base.cc:1034] running on GCE node
I20260808 02:24:37.667603 24260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:37.668596 24260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:37.669831 24260 hybrid_clock.cc:648] HybridClock initialized: now 1786155877669776 us; error 69 us; skew 500 ppm
I20260808 02:24:37.670163 24260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:37 dist-test-slave-pgkr krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155877, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:37.673425 24260 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:37.674962 24260 webserver.cc:460] Webserver started at http://127.19.231.1:33291/ using document root <none> and password file <none>
I20260808 02:24:37.675673 24260 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:37.675774 24260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:37.676043 24260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:37.679152 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "164ae061eea84619ba21bd66463b9485"
format_stamp: "Formatted at 2026-08-08 02:24:37 on dist-test-slave-pgkr"
server_key: "5544c58ddba1ddd3e5506738a8a0ef0c"
server_key_iv: "930d824bcdeeb488bc273970b266d28e"
server_key_version: "encryptionkey@0"
I20260808 02:24:37.679891 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "164ae061eea84619ba21bd66463b9485"
format_stamp: "Formatted at 2026-08-08 02:24:37 on dist-test-slave-pgkr"
server_key: "5544c58ddba1ddd3e5506738a8a0ef0c"
server_key_iv: "930d824bcdeeb488bc273970b266d28e"
server_key_version: "encryptionkey@0"
I20260808 02:24:37.685020 24260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:37.688520 24275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:37.689927 24260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:37.690063 24260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "164ae061eea84619ba21bd66463b9485"
format_stamp: "Formatted at 2026-08-08 02:24:37 on dist-test-slave-pgkr"
server_key: "5544c58ddba1ddd3e5506738a8a0ef0c"
server_key_iv: "930d824bcdeeb488bc273970b266d28e"
server_key_version: "encryptionkey@0"
I20260808 02:24:37.690231 24260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:37.709641 24260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:37.713593 24260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:37.713840 24260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:37.714776 24260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:37.716074 24260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:37.716148 24260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:37.716253 24260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:37.716298 24260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:37.751847 24260 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:44161
I20260808 02:24:37.751894 24393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:44161 every 8 connection(s)
I20260808 02:24:37.753207 24260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:37.756973 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 24260
I20260808 02:24:37.757220 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:37.758026 20380 external_mini_cluster.cc:1428] Setting key 7f6eefa7f18bf7f9cf7a4d12828ac526
Aug 08 02:24:37 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155877, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:37.774883 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:37.755896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:57807 (local address 127.19.231.62:36507)
0808 02:24:37.756342 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:37.756347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:37.757644 (+ 1297us) server_negotiation.cc:407] Connection header received
0808 02:24:37.759692 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:37.759698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:37.759781 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:37.759907 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:37.761432 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:37.762191 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:37.763802 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:37.764220 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:37.764364 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:37.768267 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:37.768276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:37.768282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:37.768329 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:37.771140 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:37.772161 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:37.772165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:37.772168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:37.772298 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:37.772867 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:37.772870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:37.772872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:37.773129 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:37.773244 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:37.774234 (+ 990us) server_negotiation.cc:299] Negotiation successful
0808 02:24:37.774397 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260808 02:24:37.776568 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:37.756557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:36507 (local address 127.19.231.1:57807)
0808 02:24:37.757471 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:37.757512 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:37.759419 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:37.760152 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:37.760160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:37.760697 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:37.761264 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:37.761281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:37.762351 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:37.762355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:37.763650 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:37.763663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:37.764460 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:37.764469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:37.764729 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:37.765490 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:37.765527 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:37.768030 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:37.771301 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:37.771314 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:37.771334 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:37.772032 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:37.772411 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:37.772419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:37.772426 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:37.772739 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:37.773277 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:37.773289 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:37.773780 (+ 491us) client_negotiation.cc:769] Sending connection context
0808 02:24:37.774134 (+ 354us) client_negotiation.cc:240] Negotiation successful
0808 02:24:37.774847 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":324,"threads_started":1}
I20260808 02:24:37.778388 24394 heartbeater.cc:347] Connected to a master server at 127.19.231.62:36507
I20260808 02:24:37.778786 24394 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:37.779677 24394 heartbeater.cc:511] Master 127.19.231.62:36507 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
I20260808 02:24:37.781910 24196 ts_manager.cc:194] Registered new tserver with Master: 164ae061eea84619ba21bd66463b9485 (127.19.231.1:44161)
I20260808 02:24:37.783579 24196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:57807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:37.813462 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:36507
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:38.150242 24400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:38.150563 24400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:38.150754 24400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:38.158070 24400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:38.158214 24400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:38.158408 24400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:38.168337 24400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:36507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:38.169698 24400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:38.171078 24400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:38.184808 24405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:38.184836 24406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:38.184855 24408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:38.185787 24400 server_base.cc:1034] running on GCE node
I20260808 02:24:38.186475 24400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:38.187575 24400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:38.188880 24400 hybrid_clock.cc:648] HybridClock initialized: now 1786155878188813 us; error 66 us; skew 500 ppm
I20260808 02:24:38.189281 24400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:38 dist-test-slave-pgkr krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:38.193073 24400 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:38.194859 24400 webserver.cc:460] Webserver started at http://127.19.231.2:40423/ using document root <none> and password file <none>
I20260808 02:24:38.195590 24400 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:38.195698 24400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:38.195964 24400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:38.198926 24400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "10dece77f9734c338ac99381f8baa012"
format_stamp: "Formatted at 2026-08-08 02:24:38 on dist-test-slave-pgkr"
server_key: "05b6187f34087b7022210db39221864b"
server_key_iv: "73e606b62d5848029fde34ce02876dc6"
server_key_version: "encryptionkey@0"
I20260808 02:24:38.199731 24400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "10dece77f9734c338ac99381f8baa012"
format_stamp: "Formatted at 2026-08-08 02:24:38 on dist-test-slave-pgkr"
server_key: "05b6187f34087b7022210db39221864b"
server_key_iv: "73e606b62d5848029fde34ce02876dc6"
server_key_version: "encryptionkey@0"
I20260808 02:24:38.205432 24400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:38.208961 24416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:38.210310 24400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 02:24:38.210481 24400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "10dece77f9734c338ac99381f8baa012"
format_stamp: "Formatted at 2026-08-08 02:24:38 on dist-test-slave-pgkr"
server_key: "05b6187f34087b7022210db39221864b"
server_key_iv: "73e606b62d5848029fde34ce02876dc6"
server_key_version: "encryptionkey@0"
I20260808 02:24:38.210742 24400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:38.243263 24400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:38.248473 24400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:38.248811 24400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:38.249861 24400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:38.251310 24400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:38.251389 24400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:38.251534 24400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:38.251587 24400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:38.287593 24400 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:38843
I20260808 02:24:38.287688 24534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:38843 every 8 connection(s)
I20260808 02:24:38.288995 24400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:38.296337 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 24400
I20260808 02:24:38.296516 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:38.296958 20380 external_mini_cluster.cc:1428] Setting key 2f9c32551e22515a080b2799b80bac61
Aug 08 02:24:38 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:38.312516 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:38.291903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:49497 (local address 127.19.231.62:36507)
0808 02:24:38.292359 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:38.292364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:38.293417 (+ 1053us) server_negotiation.cc:407] Connection header received
0808 02:24:38.295745 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:38.295751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:38.295854 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:38.295988 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:38.298034 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.299022 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.300671 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.301558 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.301722 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:38.305686 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:38.305695 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:38.305703 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:38.305753 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:38.309060 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.310008 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.310012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.310015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.310163 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.310690 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.310693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.310696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.310966 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:38.311111 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:38.312026 (+ 915us) server_negotiation.cc:299] Negotiation successful
0808 02:24:38.312216 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":208,"threads_started":1}
I20260808 02:24:38.314100 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:38.292445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:36507 (local address 127.19.231.2:49497)
0808 02:24:38.293253 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:38.293303 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:38.295459 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:38.296238 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:38.296246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:38.296989 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:38.297853 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:38.297881 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.299189 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.299195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:38.300497 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:38.300521 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.301808 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.301813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:38.302056 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:38.302863 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:38.302895 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:38.305447 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:38.309202 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:38.309214 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:38.309232 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:38.309885 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:38.310276 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:38.310283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:38.310290 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:38.310525 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:38.311101 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:38.311111 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:38.311585 (+ 474us) client_negotiation.cc:769] Sending connection context
0808 02:24:38.311948 (+ 363us) client_negotiation.cc:240] Negotiation successful
0808 02:24:38.312536 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":347,"threads_started":1}
I20260808 02:24:38.315704 24535 heartbeater.cc:347] Connected to a master server at 127.19.231.62:36507
I20260808 02:24:38.316035 24535 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:38.316802 24535 heartbeater.cc:511] Master 127.19.231.62:36507 requested a full tablet report, sending...
I20260808 02:24:38.318403 24196 ts_manager.cc:194] Registered new tserver with Master: 10dece77f9734c338ac99381f8baa012 (127.19.231.2:38843)
I20260808 02:24:38.319386 24196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:49497
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:38.353945 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:36507
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:38.697422 24541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:38.697698 24541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:38.697791 24541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:38.704821 24541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:38.704924 24541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:38.705035 24541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:38.714924 24541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:36507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:38.716208 24541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:38.717479 24541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:38.731457 24546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:38.731480 24547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:38.731927 24549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:38.733326 24541 server_base.cc:1034] running on GCE node
I20260808 02:24:38.734108 24541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:38.735154 24541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:38.736398 24541 hybrid_clock.cc:648] HybridClock initialized: now 1786155878736325 us; error 32 us; skew 500 ppm
I20260808 02:24:38.736804 24541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:38 dist-test-slave-pgkr krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:38.740140 24541 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:38.741526 24541 webserver.cc:460] Webserver started at http://127.19.231.3:37917/ using document root <none> and password file <none>
I20260808 02:24:38.742249 24541 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:38.742394 24541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:38.742797 24541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:38.745726 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "a9b0b94ebe084f46b456df766b35a6a6"
format_stamp: "Formatted at 2026-08-08 02:24:38 on dist-test-slave-pgkr"
server_key: "cff611d33c4f5813ca77f77e706b23e9"
server_key_iv: "316e82a4cc754d04267badd976c758aa"
server_key_version: "encryptionkey@0"
I20260808 02:24:38.746512 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "a9b0b94ebe084f46b456df766b35a6a6"
format_stamp: "Formatted at 2026-08-08 02:24:38 on dist-test-slave-pgkr"
server_key: "cff611d33c4f5813ca77f77e706b23e9"
server_key_iv: "316e82a4cc754d04267badd976c758aa"
server_key_version: "encryptionkey@0"
I20260808 02:24:38.751955 24541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:24:38.756553 24556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:38.758016 24541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:24:38.758140 24541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "a9b0b94ebe084f46b456df766b35a6a6"
format_stamp: "Formatted at 2026-08-08 02:24:38 on dist-test-slave-pgkr"
server_key: "cff611d33c4f5813ca77f77e706b23e9"
server_key_iv: "316e82a4cc754d04267badd976c758aa"
server_key_version: "encryptionkey@0"
I20260808 02:24:38.758309 24541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:38.772831 24541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:38.776763 24541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:38.776996 24541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:38.777940 24541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:38.779501 24541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:38.779601 24541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:38.779723 24541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:38.779778 24541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:38.786875 24394 heartbeater.cc:503] Master 127.19.231.62:36507 was elected leader, sending a full tablet report...
I20260808 02:24:38.815073 24541 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:42499
I20260808 02:24:38.815239 24671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:42499 every 8 connection(s)
I20260808 02:24:38.816506 24541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:38.818167 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 24541
I20260808 02:24:38.818398 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:38.818995 20380 external_mini_cluster.cc:1428] Setting key e5dc3bf916657239e05ddd545a4109c3
Aug 08 02:24:38 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:38.838240 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:38.819460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:38917 (local address 127.19.231.62:36507)
0808 02:24:38.820563 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:38.820572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:38.821315 (+ 743us) server_negotiation.cc:407] Connection header received
0808 02:24:38.823726 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:38.823730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:38.823796 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:38.823884 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:38.825406 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.826173 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.827922 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.828300 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.828395 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:38.832002 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:38.832006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:38.832009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:38.832041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:38.835115 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.835865 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.835875 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.835878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.835999 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.836461 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.836472 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.836475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.836742 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:38.836904 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:38.837599 (+ 695us) server_negotiation.cc:299] Negotiation successful
0808 02:24:38.837847 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":582,"threads_started":1}
I20260808 02:24:38.839277 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:38.820195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:36507 (local address 127.19.231.3:38917)
0808 02:24:38.821107 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:38.821168 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:38.823456 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:38.824115 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:38.824123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:38.824666 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:38.825259 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:38.825277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.826322 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.826326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:38.827800 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:38.827813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.828612 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.828618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:38.828818 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:38.829450 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:38.829475 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:38.831804 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:38.835270 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:38.835279 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:38.835307 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:38.835733 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:38.836118 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:38.836124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:38.836128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:38.836331 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:38.836873 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:38.836881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:38.837214 (+ 333us) client_negotiation.cc:769] Sending connection context
0808 02:24:38.837487 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 02:24:38.838042 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":422,"threads_started":1}
I20260808 02:24:38.840628 24672 heartbeater.cc:347] Connected to a master server at 127.19.231.62:36507
I20260808 02:24:38.840950 24672 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:38.841668 24672 heartbeater.cc:511] Master 127.19.231.62:36507 requested a full tablet report, sending...
I20260808 02:24:38.843137 24196 ts_manager.cc:194] Registered new tserver with Master: a9b0b94ebe084f46b456df766b35a6a6 (127.19.231.3:42499)
I20260808 02:24:38.843909 24196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:38917
I20260808 02:24:38.847126 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:24:38 dist-test-slave-pgkr krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:24:38 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:38.904264 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:38.887231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.19.231.1:44161)
0808 02:24:38.887874 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:38.887882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:38.887922 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:24:38.887984 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:38.887990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:38.888212 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:38.888341 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:38.889821 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.891297 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.892669 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.893458 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.893580 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:38.897858 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:38.897870 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:38.897886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:38.897922 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:38.901167 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.901889 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.901903 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.901906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.902036 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.902449 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.902456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.902459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.902896 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:38.903244 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:38.903592 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 02:24:38.903883 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":314,"threads_started":1}
Aug 08 02:24:38 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:38.922820 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:38.909166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.19.231.62:36507)
0808 02:24:38.909378 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:38.909387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:38.909447 (+ 60us) server_negotiation.cc:407] Connection header received
0808 02:24:38.909655 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:38.909661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:38.909737 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:38.909841 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:38.911134 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.911853 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.913041 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.913474 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.913611 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:38.917541 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:38.917546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:38.917549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:38.917580 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:38.920204 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.920824 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.920831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.920834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.920958 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.921318 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.921323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.921327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.921635 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:38.921896 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:38.922188 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 02:24:38.922424 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 02:24:38.923450 24196 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45662
I20260808 02:24:38.924515 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 24260
I20260808 02:24:38.938184 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:38.925477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.19.231.1:44161)
0808 02:24:38.925728 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:38.925737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:38.925990 (+ 253us) server_negotiation.cc:407] Connection header received
0808 02:24:38.926204 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:38.926210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:38.926286 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:38.926403 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:38.927743 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.928488 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.929826 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:38.930275 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:38.930419 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:38.932798 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:38.932812 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:38.932816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:38.932848 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:38.935454 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.936167 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.936174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.936178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.936313 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:38.936721 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:38.936727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:38.936731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:38.936992 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:38.937287 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:38.937592 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 02:24:38.937840 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:24:38.938813 24389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:39.025501 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 24260
I20260808 02:24:39.047047 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 24400
Aug 08 02:24:39 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:39.063934 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:39.048056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.19.231.2:38843)
0808 02:24:39.048649 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:39.048657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:39.048683 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:24:39.048764 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:39.048791 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:39.049021 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:39.049216 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:39.050528 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:39.051638 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:39.052961 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:39.053544 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:39.053693 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:39.057965 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:39.057976 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:39.057991 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:39.058030 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:39.060935 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:39.061657 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:39.061668 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:39.061673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:39.061818 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:39.062211 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:39.062222 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:39.062225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:39.062561 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:39.062898 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:39.063296 (+ 398us) server_negotiation.cc:299] Negotiation successful
0808 02:24:39.063571 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":277,"threads_started":1}
I20260808 02:24:39.065233 24530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:39.149276 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 24400
I20260808 02:24:39.169274 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 24541
Aug 08 02:24:39 dist-test-slave-pgkr krb5kdc[24147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:39.185613 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:39.170318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.19.231.3:42499)
0808 02:24:39.171047 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:39.171054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:39.171083 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:24:39.171145 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:39.171159 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:39.171371 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:39.171499 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:39.172757 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:39.173781 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:39.175173 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:39.175718 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:39.175816 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:39.180234 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:39.180246 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:39.180259 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:39.180293 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:39.183233 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:39.183805 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:39.183811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:39.183813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:39.183931 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:39.184255 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:39.184258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:39.184260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:39.184573 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:39.184807 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:39.185020 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 02:24:39.185220 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":420,"threads_started":1}
I20260808 02:24:39.186973 24667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:39.274530 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 24541
I20260808 02:24:39.294842 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 24160
I20260808 02:24:39.296121 24229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:39.401392 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 24160
2026-08-08T02:24:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4092 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:39 dist-test-slave-pgkr krb5kdc[24694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:39 dist-test-slave-pgkr krb5kdc[24694](info): set up 2 sockets
Aug 08 02:24:39 dist-test-slave-pgkr krb5kdc[24694](info): commencing operation
krb5kdc: starting...
W20260808 02:24:40.555475 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:40 dist-test-slave-pgkr krb5kdc[24694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:40.807910 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37369
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:41.155864 24709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:41.156213 24709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:41.156374 24709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:41.163763 24709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:41.163913 24709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:41.164018 24709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:41.164139 24709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:41.164232 24709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:41.174584 24709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:41.176075 24709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:41.177445 24709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:41.189560 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:41.189620 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:41.189622 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:41.192024 24709 server_base.cc:1034] running on GCE node
I20260808 02:24:41.192817 24709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:41.194238 24709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:41.195478 24709 hybrid_clock.cc:648] HybridClock initialized: now 1786155881195427 us; error 45 us; skew 500 ppm
I20260808 02:24:41.195860 24709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:41 dist-test-slave-pgkr krb5kdc[24694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155881, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:41.199272 24709 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:41.200452 24709 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:41.200544 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:41.200838 24709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:41.203953 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "577b521808724c56bca409ee23765a11"
format_stamp: "Formatted at 2026-08-08 02:24:41 on dist-test-slave-pgkr"
server_key: "5e1bd73e6e6594bdf661860942cda158"
server_key_iv: "71d91679cdbda8136ca503aa8eb10848"
server_key_version: "encryptionkey@0"
I20260808 02:24:41.204737 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "577b521808724c56bca409ee23765a11"
format_stamp: "Formatted at 2026-08-08 02:24:41 on dist-test-slave-pgkr"
server_key: "5e1bd73e6e6594bdf661860942cda158"
server_key_iv: "71d91679cdbda8136ca503aa8eb10848"
server_key_version: "encryptionkey@0"
I20260808 02:24:41.210158 24709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260808 02:24:41.214104 24724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:41.215656 24709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:24:41.215926 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "577b521808724c56bca409ee23765a11"
format_stamp: "Formatted at 2026-08-08 02:24:41 on dist-test-slave-pgkr"
server_key: "5e1bd73e6e6594bdf661860942cda158"
server_key_iv: "71d91679cdbda8136ca503aa8eb10848"
server_key_version: "encryptionkey@0"
I20260808 02:24:41.216199 24709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:41.233528 24709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:41.238474 24709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:41.238833 24709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:41.263103 24709 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:37369
I20260808 02:24:41.263074 24775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:37369 every 8 connection(s)
I20260808 02:24:41.264114 24709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:41.267987 24776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:41.270951 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 24709
I20260808 02:24:41.271200 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:41.271998 20380 external_mini_cluster.cc:1428] Setting key 7431fd14444fbe97dc4bac2368e78b72
I20260808 02:24:41.280640 24776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11: Bootstrap starting.
I20260808 02:24:41.283878 24776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:41.285040 24776 log.cc:824] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:24:41 dist-test-slave-pgkr krb5kdc[24694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:41.288084 24776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11: No bootstrap required, opened a new log
I20260808 02:24:41.296168 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:41.274316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.19.231.62:37369)
0808 02:24:41.275202 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:41.275215 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:41.275261 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:24:41.277107 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:41.277133 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:41.277583 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:41.278026 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:41.279799 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:41.281216 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:41.282693 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:41.283410 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:41.283670 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:41.288324 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:41.288334 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:41.288350 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:41.288391 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:41.291828 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:41.292568 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:41.292578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:41.292584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:41.292757 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:41.293184 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:41.293189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:41.293197 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:41.293799 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:41.294131 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:41.294657 (+ 526us) server_negotiation.cc:299] Negotiation successful
0808 02:24:41.295092 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":419,"threads_started":1}
I20260808 02:24:41.296478 24776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b521808724c56bca409ee23765a11" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37369 } }
I20260808 02:24:41.296870 24776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:41.296984 24776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577b521808724c56bca409ee23765a11, State: Initialized, Role: FOLLOWER
I20260808 02:24:41.297698 24776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b521808724c56bca409ee23765a11" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37369 } }
I20260808 02:24:41.298014 24776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:41.298187 24776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:41.298368 24776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:41.300974 24776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b521808724c56bca409ee23765a11" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37369 } }
I20260808 02:24:41.301415 24776 leader_election.cc:302] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 577b521808724c56bca409ee23765a11; no voters:
I20260808 02:24:41.301939 24776 leader_election.cc:288] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:41.302111 24781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:41.302947 24781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [term 1 LEADER]: Becoming Leader. State: Replica: 577b521808724c56bca409ee23765a11, State: Running, Role: LEADER
I20260808 02:24:41.303493 24781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b521808724c56bca409ee23765a11" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37369 } }
I20260808 02:24:41.304240 24776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:41.306675 24783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 577b521808724c56bca409ee23765a11. Latest consensus state: current_term: 1 leader_uuid: "577b521808724c56bca409ee23765a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b521808724c56bca409ee23765a11" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37369 } } }
I20260808 02:24:41.306674 24782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "577b521808724c56bca409ee23765a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b521808724c56bca409ee23765a11" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37369 } } }
I20260808 02:24:41.307124 24783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:41.307160 24782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:41.309232 24786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:41.313791 24786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:41.323815 24786 catalog_manager.cc:1347] Generated new cluster ID: 6dab272012a441bb916091f53b8b5a9e
I20260808 02:24:41.323925 24786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:41.338186 24786 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:41.339614 24786 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:41.346441 24786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11: Generated new TSK 0
I20260808 02:24:41.347282 24786 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:41.411000 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37369
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:41.755268 24803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:41.755587 24803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:41.755769 24803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:41.763013 24803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:41.763172 24803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:41.763350 24803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:41.773072 24803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:41.774471 24803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:41.775825 24803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:41.789567 24809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:41.789610 24808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:41.789525 24811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:41.795776 24803 server_base.cc:1034] running on GCE node
I20260808 02:24:41.796362 24803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:41.797331 24803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:41.798563 24803 hybrid_clock.cc:648] HybridClock initialized: now 1786155881798491 us; error 59 us; skew 500 ppm
I20260808 02:24:41.798906 24803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:41 dist-test-slave-pgkr krb5kdc[24694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155881, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:41.802146 24803 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:41.803335 24803 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:41.803450 24803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:41.803789 24803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:41.806922 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "897e36eef2134660a4fccae784b9e64b"
format_stamp: "Formatted at 2026-08-08 02:24:41 on dist-test-slave-pgkr"
server_key: "f38c2cbe2b275557545e80afb226ccca"
server_key_iv: "4367b323dc91b2ff9c6899d7149dc74e"
server_key_version: "encryptionkey@0"
I20260808 02:24:41.807724 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "897e36eef2134660a4fccae784b9e64b"
format_stamp: "Formatted at 2026-08-08 02:24:41 on dist-test-slave-pgkr"
server_key: "f38c2cbe2b275557545e80afb226ccca"
server_key_iv: "4367b323dc91b2ff9c6899d7149dc74e"
server_key_version: "encryptionkey@0"
I20260808 02:24:41.812723 24803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:41.816226 24817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:41.817461 24803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:24:41.817632 24803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "897e36eef2134660a4fccae784b9e64b"
format_stamp: "Formatted at 2026-08-08 02:24:41 on dist-test-slave-pgkr"
server_key: "f38c2cbe2b275557545e80afb226ccca"
server_key_iv: "4367b323dc91b2ff9c6899d7149dc74e"
server_key_version: "encryptionkey@0"
I20260808 02:24:41.817871 24803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:41.859361 24803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:41.864328 24803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:41.864667 24803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:41.865710 24803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:41.867274 24803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:41.867377 24803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:41.867561 24803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:41.867650 24803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:41.903270 24803 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:34255
I20260808 02:24:41.903342 24935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:34255 every 8 connection(s)
I20260808 02:24:41.904256 24803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:41.907017 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 24803
I20260808 02:24:41.907294 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:41.908001 20380 external_mini_cluster.cc:1428] Setting key d9a60694010d7f7d7e74aa85980ce6e0
Aug 08 02:24:41 dist-test-slave-pgkr krb5kdc[24694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155881, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:41.927753 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:41.907023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:38869 (local address 127.19.231.62:37369)
0808 02:24:41.907570 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:41.907578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:41.908788 (+ 1210us) server_negotiation.cc:407] Connection header received
0808 02:24:41.911117 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:41.911120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:41.911222 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:41.911313 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:41.913670 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:41.914395 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:41.917067 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:41.917542 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:41.917687 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:41.921919 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:41.921924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:41.921928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:41.921959 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:41.924577 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:41.925304 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:41.925309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:41.925311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:41.925432 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:41.925880 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:41.925889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:41.925891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:41.926231 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:41.926389 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:41.927147 (+ 758us) server_negotiation.cc:299] Negotiation successful
0808 02:24:41.927376 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260808 02:24:41.928773 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:41.907667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37369 (local address 127.19.231.1:38869)
0808 02:24:41.908591 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:41.908642 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:41.910791 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:41.911615 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:41.911630 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:41.912547 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:41.913463 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:41.913494 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:41.914580 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:41.914588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:41.916902 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:41.916922 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:41.917808 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:41.917819 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:41.918193 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:41.919049 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:41.919072 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:41.921688 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:41.924731 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:41.924740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:41.924755 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:41.925168 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:41.925549 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:41.925554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:41.925558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:41.925754 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:41.926361 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:41.926369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:41.926773 (+ 404us) client_negotiation.cc:769] Sending connection context
0808 02:24:41.927038 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 02:24:41.927550 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":418,"threads_started":1}
I20260808 02:24:41.930540 24936 heartbeater.cc:347] Connected to a master server at 127.19.231.62:37369
I20260808 02:24:41.930946 24936 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:41.931841 24936 heartbeater.cc:511] Master 127.19.231.62:37369 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
I20260808 02:24:41.934442 24741 ts_manager.cc:194] Registered new tserver with Master: 897e36eef2134660a4fccae784b9e64b (127.19.231.1:34255)
I20260808 02:24:41.936053 24741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:38869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:41.963912 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37369
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:42.315570 24942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:42.315968 24942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:42.316152 24942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:42.324043 24942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:42.324190 24942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:42.324358 24942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:42.334398 24942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:42.335801 24942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:42.337242 24942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:42.350961 24947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:42.351009 24948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:42.350919 24950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:42.352042 24942 server_base.cc:1034] running on GCE node
I20260808 02:24:42.352695 24942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:42.353678 24942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:42.354934 24942 hybrid_clock.cc:648] HybridClock initialized: now 1786155882354867 us; error 51 us; skew 500 ppm
I20260808 02:24:42.355280 24942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:42 dist-test-slave-pgkr krb5kdc[24694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:42.358479 24942 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:42.359673 24942 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:42.359763 24942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:42.360018 24942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:42.362906 24942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "649c2be11c2e4908b7e6006619b69464"
format_stamp: "Formatted at 2026-08-08 02:24:42 on dist-test-slave-pgkr"
server_key: "6e845345193a2d62232de32fb8a3ff2a"
server_key_iv: "8583f3454519e97e92d50191551aa370"
server_key_version: "encryptionkey@0"
I20260808 02:24:42.363642 24942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "649c2be11c2e4908b7e6006619b69464"
format_stamp: "Formatted at 2026-08-08 02:24:42 on dist-test-slave-pgkr"
server_key: "6e845345193a2d62232de32fb8a3ff2a"
server_key_iv: "8583f3454519e97e92d50191551aa370"
server_key_version: "encryptionkey@0"
I20260808 02:24:42.368561 24942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:42.372008 24957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:42.373306 24942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:42.373447 24942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "649c2be11c2e4908b7e6006619b69464"
format_stamp: "Formatted at 2026-08-08 02:24:42 on dist-test-slave-pgkr"
server_key: "6e845345193a2d62232de32fb8a3ff2a"
server_key_iv: "8583f3454519e97e92d50191551aa370"
server_key_version: "encryptionkey@0"
I20260808 02:24:42.373617 24942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:42.387852 24942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:42.392117 24942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:42.392366 24942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:42.393311 24942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:42.394747 24942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:42.394834 24942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:42.394912 24942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:42.394968 24942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:42.429556 24942 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:42669
I20260808 02:24:42.429677 25079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:42669 every 8 connection(s)
I20260808 02:24:42.430451 24942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:42.438284 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 24942
I20260808 02:24:42.438545 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:42.439055 20380 external_mini_cluster.cc:1428] Setting key 44ae796f331007480907c9059289d500
Aug 08 02:24:42 dist-test-slave-pgkr krb5kdc[24694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:42.452857 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:42.433149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:47829 (local address 127.19.231.62:37369)
0808 02:24:42.433722 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:42.433727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:42.434446 (+ 719us) server_negotiation.cc:407] Connection header received
0808 02:24:42.436545 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:42.436553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:42.436645 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:42.436774 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:42.438697 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.439553 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.441545 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.441985 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.442127 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:42.445723 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:42.445728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:42.445732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:42.445762 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:42.448964 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:42.449992 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:42.449999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:42.450004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:42.450201 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:42.450755 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:42.450763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:42.450767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:42.451185 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:42.451353 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:42.452119 (+ 766us) server_negotiation.cc:299] Negotiation successful
0808 02:24:42.452386 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":289,"threads_started":1}
I20260808 02:24:42.453883 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:42.433443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37369 (local address 127.19.231.2:47829)
0808 02:24:42.434258 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:42.434306 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:42.436227 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:42.437038 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:42.437047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:42.437604 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:42.438444 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:42.438475 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.439736 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.439740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:42.441356 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:42.441375 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.442228 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.442236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:42.442455 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:42.443265 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:42.443296 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:42.445488 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:42.449125 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:42.449137 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:42.449156 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:42.449856 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:42.450333 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:42.450354 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:42.450358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:42.450574 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:42.451326 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:42.451332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:42.451692 (+ 360us) client_negotiation.cc:769] Sending connection context
0808 02:24:42.451997 (+ 305us) client_negotiation.cc:240] Negotiation successful
0808 02:24:42.452596 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":324,"threads_started":1}
I20260808 02:24:42.455864 25080 heartbeater.cc:347] Connected to a master server at 127.19.231.62:37369
I20260808 02:24:42.456218 25080 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:42.456928 25080 heartbeater.cc:511] Master 127.19.231.62:37369 requested a full tablet report, sending...
I20260808 02:24:42.458487 24741 ts_manager.cc:194] Registered new tserver with Master: 649c2be11c2e4908b7e6006619b69464 (127.19.231.2:42669)
I20260808 02:24:42.459528 24741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:47829
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:42.494989 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37369
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:42.833828 25086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:42.834177 25086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:42.834357 25086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:42.841797 25086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:42.841943 25086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:42.842136 25086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:42.852190 25086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:37369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:42.853603 25086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:42.855074 25086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:42.868254 25091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:42.868194 25094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:42.868219 25092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:42.869077 25086 server_base.cc:1034] running on GCE node
I20260808 02:24:42.869743 25086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:42.870829 25086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:42.872067 25086 hybrid_clock.cc:648] HybridClock initialized: now 1786155882872004 us; error 52 us; skew 500 ppm
I20260808 02:24:42.872409 25086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:42 dist-test-slave-pgkr krb5kdc[24694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:42.876087 25086 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:42.877290 25086 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:42.877379 25086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:42.877687 25086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:42.880749 25086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "76a440c6367146aeb9a115c2828cda9c"
format_stamp: "Formatted at 2026-08-08 02:24:42 on dist-test-slave-pgkr"
server_key: "84de3ac74032208e93f113f3ea143c66"
server_key_iv: "8815686252a103ad5848c6c5dd3ceb20"
server_key_version: "encryptionkey@0"
I20260808 02:24:42.881502 25086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "76a440c6367146aeb9a115c2828cda9c"
format_stamp: "Formatted at 2026-08-08 02:24:42 on dist-test-slave-pgkr"
server_key: "84de3ac74032208e93f113f3ea143c66"
server_key_iv: "8815686252a103ad5848c6c5dd3ceb20"
server_key_version: "encryptionkey@0"
I20260808 02:24:42.886731 25086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260808 02:24:42.890278 25100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:42.891727 25086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:24:42.891863 25086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "76a440c6367146aeb9a115c2828cda9c"
format_stamp: "Formatted at 2026-08-08 02:24:42 on dist-test-slave-pgkr"
server_key: "84de3ac74032208e93f113f3ea143c66"
server_key_iv: "8815686252a103ad5848c6c5dd3ceb20"
server_key_version: "encryptionkey@0"
I20260808 02:24:42.892047 25086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:42.907481 25086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:42.911814 25086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:42.912101 25086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:42.913100 25086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:42.914489 25086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:42.914579 25086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:42.914754 25086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:42.914851 25086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:42.939177 24936 heartbeater.cc:503] Master 127.19.231.62:37369 was elected leader, sending a full tablet report...
I20260808 02:24:42.950244 25086 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:37079
I20260808 02:24:42.950332 25216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:37079 every 8 connection(s)
I20260808 02:24:42.951188 25086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:42.958869 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 25086
I20260808 02:24:42.959146 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:42.959698 20380 external_mini_cluster.cc:1428] Setting key aef410ed6a180aa4b9db39d9c03e164c
Aug 08 02:24:42 dist-test-slave-pgkr krb5kdc[24694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:42.971553 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:42.953878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:59285 (local address 127.19.231.62:37369)
0808 02:24:42.954337 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:42.954343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:42.955460 (+ 1117us) server_negotiation.cc:407] Connection header received
0808 02:24:42.957238 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:42.957242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:42.957310 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:42.957407 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:42.959001 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.960225 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.962056 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.962548 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.962752 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:42.966199 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:42.966204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:42.966207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:42.966240 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:42.968758 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:42.969446 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:42.969450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:42.969452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:42.969568 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:42.969949 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:42.969952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:42.969954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:42.970215 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:42.970333 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:42.971094 (+ 761us) server_negotiation.cc:299] Negotiation successful
0808 02:24:42.971280 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":202,"threads_started":1}
I20260808 02:24:42.972604 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:42.954527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37369 (local address 127.19.231.3:59285)
0808 02:24:42.955292 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:42.955337 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:42.956967 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:42.957613 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:42.957621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:42.958164 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:42.958814 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:42.958833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.960389 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.960394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:42.961888 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:42.961905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.962855 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.962864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:42.963112 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:42.963814 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:42.963842 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:42.966000 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:42.968885 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:42.968892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:42.968904 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:42.969342 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:42.969683 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:42.969687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:42.969689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:42.969859 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:42.970327 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:42.970333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:42.970722 (+ 389us) client_negotiation.cc:769] Sending connection context
0808 02:24:42.970986 (+ 264us) client_negotiation.cc:240] Negotiation successful
0808 02:24:42.971375 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":273,"threads_started":1}
I20260808 02:24:42.974186 25217 heartbeater.cc:347] Connected to a master server at 127.19.231.62:37369
I20260808 02:24:42.974536 25217 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:42.975323 25217 heartbeater.cc:511] Master 127.19.231.62:37369 requested a full tablet report, sending...
I20260808 02:24:42.976641 24741 ts_manager.cc:194] Registered new tserver with Master: 76a440c6367146aeb9a115c2828cda9c (127.19.231.3:37079)
I20260808 02:24:42.977501 24741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:59285
I20260808 02:24:42.987332 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:24:43.004472 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:42.991983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.19.231.62:37369)
0808 02:24:42.992154 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:42.992159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:42.992672 (+ 513us) server_negotiation.cc:407] Connection header received
0808 02:24:42.992807 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:42.992811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:42.992870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:42.992951 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:42.994358 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.995128 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.996547 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:42.996977 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:42.997073 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:42.999473 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:42.999478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:42.999481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:42.999509 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:43.002131 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.002843 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.002847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.002849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.002964 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.003290 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.003293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.003295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.003586 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:43.003785 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.004056 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.004236 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:24:43.008448 24741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:43.015401 24741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:43.039041 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.028926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.19.231.1:34255)
0808 02:24:43.029700 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.029711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.030520 (+ 809us) server_negotiation.cc:407] Connection header received
0808 02:24:43.030667 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.030678 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.030899 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.031096 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:43.032250 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.033779 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.036497 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.037867 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.038040 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.038281 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.038546 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.038732 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":350,"threads_started":1}
I20260808 02:24:43.039036 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.029071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:34255 (local address 127.0.0.1:52634)
0808 02:24:43.030106 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:43.030145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:43.030314 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:43.031365 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:43.031369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:43.031394 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:43.032006 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.032028 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.033946 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.033950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.036346 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.036358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.038109 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.038116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.038334 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.038376 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 02:24:43.038480 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:24:43.038609 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":28,"thread_start_us":615,"threads_started":1}
I20260808 02:24:43.039245 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.028082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.19.231.2:42669)
0808 02:24:43.029049 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.029063 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.030253 (+ 1190us) server_negotiation.cc:407] Connection header received
0808 02:24:43.030586 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.030594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.030873 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.031039 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:43.031997 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.033448 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.035779 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.037150 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.037290 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.037489 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.038730 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.038908 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":628,"threads_started":1}
I20260808 02:24:43.039660 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.027865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:42669 (local address 127.0.0.1:58838)
0808 02:24:43.030106 (+ 2241us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:43.030145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:43.030368 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:43.031206 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:43.031213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:43.031247 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:43.031824 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.031837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.033614 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.033619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.035630 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.035643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.037678 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.037684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.037972 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.038030 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 02:24:43.038160 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 02:24:43.038365 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1983,"mutex_wait_us":118,"thread_start_us":1888,"threads_started":1}
I20260808 02:24:43.041318 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.031363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:37079 (local address 127.0.0.1:52878)
0808 02:24:43.032431 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:43.032449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:43.032567 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:43.033566 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:43.033573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:43.033605 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:43.034152 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.034168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.036177 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.036183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.038395 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.038408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.040505 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.040512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.040676 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.040707 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:24:43.040808 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:24:43.040925 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":23,"thread_start_us":789,"threads_started":1}
I20260808 02:24:43.041376 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.030853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.19.231.3:37079)
0808 02:24:43.031702 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.031711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.032555 (+ 844us) server_negotiation.cc:407] Connection header received
0808 02:24:43.032726 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.032734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.032936 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.033119 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:43.034391 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.036025 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.038589 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.040300 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.040442 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.040671 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.040867 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.041019 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":662,"threads_started":1}
I20260808 02:24:43.043147 24866 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5556fb14f845c284d168e7e8d27e39 (DEFAULT_TABLE table=test-table [id=394dbf6a53f449a29966181e46d45023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:43.043641 25007 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5556fb14f845c284d168e7e8d27e39 (DEFAULT_TABLE table=test-table [id=394dbf6a53f449a29966181e46d45023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:43.044765 24866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5556fb14f845c284d168e7e8d27e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:43.044974 25007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5556fb14f845c284d168e7e8d27e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:43.044816 25150 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5556fb14f845c284d168e7e8d27e39 (DEFAULT_TABLE table=test-table [id=394dbf6a53f449a29966181e46d45023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:43.046128 25150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5556fb14f845c284d168e7e8d27e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:43.056653 25238 tablet_bootstrap.cc:492] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Bootstrap starting.
I20260808 02:24:43.059631 25239 tablet_bootstrap.cc:492] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Bootstrap starting.
I20260808 02:24:43.060503 25238 tablet_bootstrap.cc:654] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:43.063300 25239 tablet_bootstrap.cc:654] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:43.063405 25238 log.cc:824] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:43.064670 25239 log.cc:824] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:43.064989 25240 tablet_bootstrap.cc:492] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Bootstrap starting.
I20260808 02:24:43.067325 25238 tablet_bootstrap.cc:492] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: No bootstrap required, opened a new log
I20260808 02:24:43.067637 25238 ts_tablet_manager.cc:1397] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 02:24:43.067822 25239 tablet_bootstrap.cc:492] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: No bootstrap required, opened a new log
I20260808 02:24:43.068101 25239 ts_tablet_manager.cc:1397] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260808 02:24:43.069613 25240 tablet_bootstrap.cc:654] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:43.070791 25240 log.cc:824] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:43.074703 25240 tablet_bootstrap.cc:492] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: No bootstrap required, opened a new log
I20260808 02:24:43.074952 25240 ts_tablet_manager.cc:1397] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 02:24:43.078711 25238 raft_consensus.cc:348] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.078706 25239 raft_consensus.cc:348] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.079152 25238 raft_consensus.cc:374] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:43.079152 25239 raft_consensus.cc:374] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:43.079263 25239 raft_consensus.cc:729] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897e36eef2134660a4fccae784b9e64b, State: Initialized, Role: FOLLOWER
I20260808 02:24:43.079263 25238 raft_consensus.cc:729] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649c2be11c2e4908b7e6006619b69464, State: Initialized, Role: FOLLOWER
I20260808 02:24:43.079878 25238 consensus_queue.cc:260] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.079878 25239 consensus_queue.cc:260] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.081489 25080 heartbeater.cc:503] Master 127.19.231.62:37369 was elected leader, sending a full tablet report...
I20260808 02:24:43.082432 25238 ts_tablet_manager.cc:1428] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260808 02:24:43.083025 25239 ts_tablet_manager.cc:1428] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260808 02:24:43.085440 25240 raft_consensus.cc:348] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.085933 25240 raft_consensus.cc:374] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:43.086048 25240 raft_consensus.cc:729] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a440c6367146aeb9a115c2828cda9c, State: Initialized, Role: FOLLOWER
I20260808 02:24:43.086751 25240 consensus_queue.cc:260] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.087924 25217 heartbeater.cc:503] Master 127.19.231.62:37369 was elected leader, sending a full tablet report...
I20260808 02:24:43.088490 25240 ts_tablet_manager.cc:1428] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260808 02:24:43.156893 24937 tablet.cc:2369] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:24:43.182713 25081 tablet.cc:2369] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:43.197516 25245 raft_consensus.cc:482] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:43.197906 25245 raft_consensus.cc:504] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.199834 25245 leader_election.cc:288] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649c2be11c2e4908b7e6006619b69464 (127.19.231.2:42669), 76a440c6367146aeb9a115c2828cda9c (127.19.231.3:37079)
W20260808 02:24:43.202821 25218 tablet.cc:2369] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:43.207871 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.200430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:46409 (local address 127.19.231.2:42669)
0808 02:24:43.200718 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.200726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.201047 (+ 321us) server_negotiation.cc:407] Connection header received
0808 02:24:43.201315 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.201322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.201406 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.201475 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:43.202386 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.203354 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.205844 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.207053 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.207210 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.207291 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.207519 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.207661 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 02:24:43.208088 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.200135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:42669 (local address 127.19.231.1:46409)
0808 02:24:43.200882 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:43.200923 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:43.201114 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:43.201684 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:43.201687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:43.201735 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:43.202218 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.202238 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.203482 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.203489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.205684 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.205704 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.207243 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.207247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.207360 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.207385 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:24:43.207464 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 02:24:43.207619 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":29,"thread_start_us":387,"threads_started":1}
I20260808 02:24:43.208463 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.201274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:53315 (local address 127.19.231.3:37079)
0808 02:24:43.201477 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.201483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.201760 (+ 277us) server_negotiation.cc:407] Connection header received
0808 02:24:43.201909 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.201915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.201988 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.202106 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:24:43.202763 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.203871 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.206146 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.207428 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.207625 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.207689 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.208063 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.208227 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:24:43.209187 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.200973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:37079 (local address 127.19.231.1:53315)
0808 02:24:43.201640 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:43.201670 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:43.201777 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:43.202115 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:43.202118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:43.202139 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:24:43.202547 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.202558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.204013 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.204018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.206004 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:43.206015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.207669 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.207674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:43.207860 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.207889 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:24:43.208645 (+ 756us) client_negotiation.cc:240] Negotiation successful
0808 02:24:43.208799 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":1,"thread_start_us":333,"threads_started":1}
I20260808 02:24:43.209214 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5556fb14f845c284d168e7e8d27e39" candidate_uuid: "897e36eef2134660a4fccae784b9e64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649c2be11c2e4908b7e6006619b69464" is_pre_election: true
I20260808 02:24:43.209820 25031 raft_consensus.cc:2454] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897e36eef2134660a4fccae784b9e64b in term 0.
I20260808 02:24:43.210394 25170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5556fb14f845c284d168e7e8d27e39" candidate_uuid: "897e36eef2134660a4fccae784b9e64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a440c6367146aeb9a115c2828cda9c" is_pre_election: true
I20260808 02:24:43.210588 24819 leader_election.cc:302] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649c2be11c2e4908b7e6006619b69464, 897e36eef2134660a4fccae784b9e64b; no voters:
I20260808 02:24:43.210949 25170 raft_consensus.cc:2454] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897e36eef2134660a4fccae784b9e64b in term 0.
I20260808 02:24:43.211148 25245 raft_consensus.cc:2781] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:43.211285 25245 raft_consensus.cc:482] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:43.211416 25245 raft_consensus.cc:3037] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:43.213735 25245 raft_consensus.cc:504] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.214311 25245 leader_election.cc:288] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [CANDIDATE]: Term 1 election: Requested vote from peers 649c2be11c2e4908b7e6006619b69464 (127.19.231.2:42669), 76a440c6367146aeb9a115c2828cda9c (127.19.231.3:37079)
I20260808 02:24:43.214821 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5556fb14f845c284d168e7e8d27e39" candidate_uuid: "897e36eef2134660a4fccae784b9e64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649c2be11c2e4908b7e6006619b69464"
I20260808 02:24:43.214843 25170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5556fb14f845c284d168e7e8d27e39" candidate_uuid: "897e36eef2134660a4fccae784b9e64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a440c6367146aeb9a115c2828cda9c"
I20260808 02:24:43.215044 25170 raft_consensus.cc:3037] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:43.215062 25031 raft_consensus.cc:3037] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:43.218362 25031 raft_consensus.cc:2454] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897e36eef2134660a4fccae784b9e64b in term 1.
I20260808 02:24:43.218361 25170 raft_consensus.cc:2454] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897e36eef2134660a4fccae784b9e64b in term 1.
I20260808 02:24:43.218791 24819 leader_election.cc:302] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76a440c6367146aeb9a115c2828cda9c, 897e36eef2134660a4fccae784b9e64b; no voters:
I20260808 02:24:43.219111 25245 raft_consensus.cc:2781] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:43.219558 25245 raft_consensus.cc:686] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 1 LEADER]: Becoming Leader. State: Replica: 897e36eef2134660a4fccae784b9e64b, State: Running, Role: LEADER
I20260808 02:24:43.220053 25245 consensus_queue.cc:237] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } }
I20260808 02:24:43.225215 24740 catalog_manager.cc:5495] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b reported cstate change: term changed from 0 to 1, leader changed from <none> to 897e36eef2134660a4fccae784b9e64b (127.19.231.1). New cstate: current_term: 1 leader_uuid: "897e36eef2134660a4fccae784b9e64b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "897e36eef2134660a4fccae784b9e64b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34255 } health_report { overall_health: HEALTHY } } }
I20260808 02:24:43.283244 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.277433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.19.231.1:34255)
0808 02:24:43.277616 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.277621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.277641 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:24:43.277848 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.277852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.277932 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.278098 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:24:43.278491 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.279593 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.280980 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.281394 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.281542 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.282018 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:24:43.282547 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:24:43.282768 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.282879 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.282964 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:24:43.291833 25170 raft_consensus.cc:1261] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 1 FOLLOWER]: Refusing update from remote peer 897e36eef2134660a4fccae784b9e64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:43.292049 25031 raft_consensus.cc:1261] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 1 FOLLOWER]: Refusing update from remote peer 897e36eef2134660a4fccae784b9e64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:43.292876 25249 consensus_queue.cc:1035] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a440c6367146aeb9a115c2828cda9c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 37079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:43.293277 25245 consensus_queue.cc:1035] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649c2be11c2e4908b7e6006619b69464" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:43.305279 25252 mvcc.cc:205] Tried to move back new op lower bound from 7316094497952395264 to 7316094497673437184. Current Snapshot: MvccSnapshot[applied={T|T < 7316094497952395264}]
I20260808 02:24:43.326507 24740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:43.326872 24740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:43.330775 24740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 577b521808724c56bca409ee23765a11: Sending DeleteTablet for 3 replicas of tablet 9a5556fb14f845c284d168e7e8d27e39
I20260808 02:24:43.331974 25007 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5556fb14f845c284d168e7e8d27e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:43 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:58838
I20260808 02:24:43.332015 25150 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5556fb14f845c284d168e7e8d27e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:43 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:52878
I20260808 02:24:43.332240 24866 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5556fb14f845c284d168e7e8d27e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:43 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:52634
I20260808 02:24:43.334368 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 24803
I20260808 02:24:43.347635 25264 tablet_replica.cc:330] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: stopping tablet replica
I20260808 02:24:43.349115 25264 raft_consensus.cc:2229] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:43.351078 25264 raft_consensus.cc:2258] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:43.353914 25263 tablet_replica.cc:330] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: stopping tablet replica
I20260808 02:24:43.354405 25263 raft_consensus.cc:2229] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:43.354954 25263 raft_consensus.cc:2258] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:43.355829 25262 tablet_replica.cc:330] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: stopping tablet replica
I20260808 02:24:43.356243 25262 raft_consensus.cc:2229] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:43.356333 25264 ts_tablet_manager.cc:1905] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:43.356760 25262 raft_consensus.cc:2258] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:43.359004 25262 ts_tablet_manager.cc:1905] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 02:24:43 dist-test-slave-pgkr krb5kdc[24694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:43.364982 25264 ts_tablet_manager.cc:1918] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:43.365446 25264 log.cc:1196] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/9a5556fb14f845c284d168e7e8d27e39
I20260808 02:24:43.366103 25262 ts_tablet_manager.cc:1918] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:43.366153 25264 ts_tablet_manager.cc:1939] T 9a5556fb14f845c284d168e7e8d27e39 P 897e36eef2134660a4fccae784b9e64b: Deleting consensus metadata
I20260808 02:24:43.366279 25262 log.cc:1196] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/9a5556fb14f845c284d168e7e8d27e39
I20260808 02:24:43.366878 25262 ts_tablet_manager.cc:1939] T 9a5556fb14f845c284d168e7e8d27e39 P 649c2be11c2e4908b7e6006619b69464: Deleting consensus metadata
I20260808 02:24:43.368458 24728 catalog_manager.cc:4841] TS 897e36eef2134660a4fccae784b9e64b (127.19.231.1:34255): tablet 9a5556fb14f845c284d168e7e8d27e39 (table test-table [id=394dbf6a53f449a29966181e46d45023]) successfully deleted
I20260808 02:24:43.368897 24726 catalog_manager.cc:4841] TS 649c2be11c2e4908b7e6006619b69464 (127.19.231.2:42669): tablet 9a5556fb14f845c284d168e7e8d27e39 (table test-table [id=394dbf6a53f449a29966181e46d45023]) successfully deleted
I20260808 02:24:43.370572 25263 ts_tablet_manager.cc:1905] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:43.371112 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.336440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.19.231.1:34255)
0808 02:24:43.336694 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.336704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.347836 (+ 11132us) server_negotiation.cc:407] Connection header received
0808 02:24:43.348145 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.348152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.348265 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.348347 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:43.350198 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.351562 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.355672 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.356319 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.356432 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.362118 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:43.362138 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:43.362159 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:43.362208 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:43.366979 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.367981 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.367990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.367993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.368183 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.368701 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.368705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.368708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.369681 (+ 973us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:43.369900 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.370400 (+ 500us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.370690 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:24:43.371789 24931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:43.377637 25263 ts_tablet_manager.cc:1918] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:43.377859 25263 log.cc:1196] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/9a5556fb14f845c284d168e7e8d27e39
I20260808 02:24:43.378448 25263 ts_tablet_manager.cc:1939] T 9a5556fb14f845c284d168e7e8d27e39 P 76a440c6367146aeb9a115c2828cda9c: Deleting consensus metadata
I20260808 02:24:43.380069 24726 catalog_manager.cc:4841] TS 76a440c6367146aeb9a115c2828cda9c (127.19.231.3:37079): tablet 9a5556fb14f845c284d168e7e8d27e39 (table test-table [id=394dbf6a53f449a29966181e46d45023]) successfully deleted
I20260808 02:24:43.472465 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 24803
I20260808 02:24:43.495589 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 24942
Aug 08 02:24:43 dist-test-slave-pgkr krb5kdc[24694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:43.511302 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.496603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.19.231.2:42669)
0808 02:24:43.496806 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.496815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.496839 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:24:43.497059 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.497065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.497155 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.497264 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:43.498586 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.499393 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.500677 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.501088 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.501224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.505820 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:43.505836 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:43.505853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:43.505891 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:43.508815 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.509449 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.509456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.509458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.509575 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.509923 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.509926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.509928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.510309 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:43.510527 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.510861 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.511032 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:24:43.512084 25075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:43.602742 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 24942
I20260808 02:24:43.624114 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 25086
Aug 08 02:24:43 dist-test-slave-pgkr krb5kdc[24694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:43.639268 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:43.624895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.19.231.3:37079)
0808 02:24:43.625064 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:43.625068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:43.625088 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:24:43.625228 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:43.625233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:43.625317 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:43.625418 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:43.626769 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.627468 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.628683 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:43.629059 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:43.629148 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:43.633529 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:43.633545 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:43.633560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:43.633594 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:43.636961 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.637523 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.637530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.637533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.637658 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:43.637979 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:43.637982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:43.637984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:43.638323 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:43.638540 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:43.638853 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 02:24:43.639014 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:24:43.639911 25212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:43.729965 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 25086
I20260808 02:24:43.752983 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 24709
I20260808 02:24:43.754230 24771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:43.865841 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 24709
2026-08-08T02:24:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4464 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T02:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:43Z Disabled control of system clock
I20260808 02:24:43.947638 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:35753
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:35753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:24:44.290045 25273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:44.290341 25273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:44.290429 25273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:44.297653 25273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:44.297775 25273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:44.297816 25273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:44.297868 25273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:44.297911 25273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:44.307273 25273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:35753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:35753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:44.308565 25273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:44.310443 25273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:44.322422 25278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:44.322515 25280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:44.322782 25282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:44.323426 25273 server_base.cc:1034] running on GCE node
I20260808 02:24:44.324254 25273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:44.325642 25273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:44.326882 25273 hybrid_clock.cc:648] HybridClock initialized: now 1786155884326827 us; error 42 us; skew 500 ppm
I20260808 02:24:44.327255 25273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:44.329640 25273 webserver.cc:460] Webserver started at http://127.19.231.62:35567/ using document root <none> and password file <none>
I20260808 02:24:44.330370 25273 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:44.330506 25273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:44.330901 25273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:44.333897 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "728cbb8ec25c44d19210748db4fafae8"
format_stamp: "Formatted at 2026-08-08 02:24:44 on dist-test-slave-pgkr"
server_key: "987f9522d8efd47414dbec35c4c7c412"
server_key_iv: "b7baef4d23149b4c3e1884f67eaef855"
server_key_version: "encryptionkey@0"
I20260808 02:24:44.334802 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "728cbb8ec25c44d19210748db4fafae8"
format_stamp: "Formatted at 2026-08-08 02:24:44 on dist-test-slave-pgkr"
server_key: "987f9522d8efd47414dbec35c4c7c412"
server_key_iv: "b7baef4d23149b4c3e1884f67eaef855"
server_key_version: "encryptionkey@0"
I20260808 02:24:44.340229 25273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 02:24:44.343776 25288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:44.345238 25273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:24:44.345428 25273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "728cbb8ec25c44d19210748db4fafae8"
format_stamp: "Formatted at 2026-08-08 02:24:44 on dist-test-slave-pgkr"
server_key: "987f9522d8efd47414dbec35c4c7c412"
server_key_iv: "b7baef4d23149b4c3e1884f67eaef855"
server_key_version: "encryptionkey@0"
I20260808 02:24:44.345675 25273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:44.351605 25273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:44.352416 25273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:44.352732 25273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:44.371753 25273 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:35753
I20260808 02:24:44.371788 25339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:35753 every 8 connection(s)
I20260808 02:24:44.373176 25273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:44.377161 25340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:44.380784 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 25273
I20260808 02:24:44.380999 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:44.381912 20380 external_mini_cluster.cc:1428] Setting key b255bf08f2c5fe5e3ef1c61feeedee38
I20260808 02:24:44.390563 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:44.384188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.19.231.62:35753)
0808 02:24:44.385273 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:44.385286 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:44.385342 (+ 56us) server_negotiation.cc:407] Connection header received
0808 02:24:44.387029 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:44.387053 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:44.387492 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:44.387843 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:44.388933 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:44.388939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:44.388951 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:44.388969 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:44.389027 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:44.389044 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:44.389217 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:44.389369 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 02:24:44.389671 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":381,"threads_started":1}
I20260808 02:24:44.391183 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8: Bootstrap starting.
I20260808 02:24:44.394181 25340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:44.395251 25340 log.cc:824] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:44.398028 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8: No bootstrap required, opened a new log
I20260808 02:24:44.405242 25340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728cbb8ec25c44d19210748db4fafae8" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35753 } }
I20260808 02:24:44.405579 25340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:44.405709 25340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728cbb8ec25c44d19210748db4fafae8, State: Initialized, Role: FOLLOWER
I20260808 02:24:44.406284 25340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728cbb8ec25c44d19210748db4fafae8" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35753 } }
I20260808 02:24:44.406512 25340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:44.406669 25340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:44.406852 25340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:44.408996 25340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728cbb8ec25c44d19210748db4fafae8" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35753 } }
I20260808 02:24:44.409505 25340 leader_election.cc:302] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 728cbb8ec25c44d19210748db4fafae8; no voters:
I20260808 02:24:44.410137 25340 leader_election.cc:288] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:44.410279 25346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:44.411173 25346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [term 1 LEADER]: Becoming Leader. State: Replica: 728cbb8ec25c44d19210748db4fafae8, State: Running, Role: LEADER
I20260808 02:24:44.411718 25346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728cbb8ec25c44d19210748db4fafae8" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35753 } }
I20260808 02:24:44.412534 25340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:44.415067 25349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 728cbb8ec25c44d19210748db4fafae8. Latest consensus state: current_term: 1 leader_uuid: "728cbb8ec25c44d19210748db4fafae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728cbb8ec25c44d19210748db4fafae8" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35753 } } }
I20260808 02:24:44.415581 25349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:44.415963 25348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "728cbb8ec25c44d19210748db4fafae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "728cbb8ec25c44d19210748db4fafae8" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35753 } } }
I20260808 02:24:44.416427 25348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8 [sys.catalog]: This master's current role is: LEADER
W20260808 02:24:44.423909 25363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:24:44.424036 25363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:24:44.424196 25356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:44.425846 25356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:44.434111 25356 catalog_manager.cc:1347] Generated new cluster ID: 85427f2a6001475ebb5a946df9eba38c
I20260808 02:24:44.434257 25356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:44.453438 25356 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:44.453897 25356 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:44.468185 25356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8: Generated new TSK 0
I20260808 02:24:44.469014 25356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:24:44.476325 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35753
--builtin_ntp_servers=127.19.231.20:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 02:24:44.845149 25366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:44.845486 25366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:44.845628 25366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:44.853011 25366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:44.853114 25366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:44.853233 25366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:44.862360 25366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:44.863715 25366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:44.865492 25366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:44.879281 25374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:44.879393 25373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:44.879393 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:44.880539 25366 server_base.cc:1034] running on GCE node
I20260808 02:24:44.881255 25366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:44.882309 25366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:44.883601 25366 hybrid_clock.cc:648] HybridClock initialized: now 1786155884883523 us; error 63 us; skew 500 ppm
I20260808 02:24:44.883950 25366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:44.886350 25366 webserver.cc:460] Webserver started at http://127.19.231.1:42753/ using document root <none> and password file <none>
I20260808 02:24:44.887133 25366 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:44.887284 25366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:44.887639 25366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:44.890528 25366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "fec00b588f884a8e895617c476929794"
format_stamp: "Formatted at 2026-08-08 02:24:44 on dist-test-slave-pgkr"
server_key: "fdc6ea8100e64d0adf47e19ed5e12eaf"
server_key_iv: "c5b94c9134a94b6ff210f8feb195cb53"
server_key_version: "encryptionkey@0"
I20260808 02:24:44.891571 25366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "fec00b588f884a8e895617c476929794"
format_stamp: "Formatted at 2026-08-08 02:24:44 on dist-test-slave-pgkr"
server_key: "fdc6ea8100e64d0adf47e19ed5e12eaf"
server_key_iv: "c5b94c9134a94b6ff210f8feb195cb53"
server_key_version: "encryptionkey@0"
I20260808 02:24:44.896721 25366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 02:24:44.900089 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:44.901408 25366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:24:44.901614 25366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "fec00b588f884a8e895617c476929794"
format_stamp: "Formatted at 2026-08-08 02:24:44 on dist-test-slave-pgkr"
server_key: "fdc6ea8100e64d0adf47e19ed5e12eaf"
server_key_iv: "c5b94c9134a94b6ff210f8feb195cb53"
server_key_version: "encryptionkey@0"
I20260808 02:24:44.901837 25366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:44.907548 25366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:44.908308 25366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:44.908586 25366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:44.909461 25366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:44.910931 25366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:44.911053 25366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:44.911186 25366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:44.911275 25366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:44.945890 25366 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:32835
I20260808 02:24:44.945941 25499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:32835 every 8 connection(s)
I20260808 02:24:44.947340 25366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:44.954887 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 25366
I20260808 02:24:44.955108 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:44.955962 20380 external_mini_cluster.cc:1428] Setting key d7ecc0ab2acc6720f56dcbb4ffcb0485
I20260808 02:24:44.956103 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:44.950115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:53915 (local address 127.19.231.62:35753)
0808 02:24:44.950585 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:44.950590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:44.951788 (+ 1198us) server_negotiation.cc:407] Connection header received
0808 02:24:44.953547 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:44.953553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:44.953633 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:44.953755 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:44.955322 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:44.955327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:44.955331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:44.955350 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:44.955397 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:44.955409 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:44.955529 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:44.955779 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 02:24:44.955832 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
I20260808 02:24:44.957381 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:44.950733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:35753 (local address 127.19.231.1:53915)
0808 02:24:44.951621 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:44.951672 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:44.953266 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:44.953985 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:44.953993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:44.954120 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:44.954953 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:44.954958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:44.955014 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:44.955137 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:44.955526 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:44.955541 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:44.955549 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:24:44.955865 (+ 316us) client_negotiation.cc:240] Negotiation successful
0808 02:24:44.956151 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":264,"threads_started":1}
I20260808 02:24:44.958968 25500 heartbeater.cc:347] Connected to a master server at 127.19.231.62:35753
I20260808 02:24:44.959199 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35753
--builtin_ntp_servers=127.19.231.20:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 02:24:44.959287 25500 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:44.959893 25500 heartbeater.cc:511] Master 127.19.231.62:35753 requested a full tablet report, sending...
I20260808 02:24:44.962109 25305 ts_manager.cc:194] Registered new tserver with Master: fec00b588f884a8e895617c476929794 (127.19.231.1:32835)
W20260808 02:24:45.299108 25504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:45.299408 25504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:45.299553 25504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:45.307622 25504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:45.307734 25504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:45.307919 25504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:45.317283 25504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:45.318750 25504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:45.320595 25504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:45.334015 25509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:45.334043 25510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:45.334950 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:45.335606 25504 server_base.cc:1034] running on GCE node
I20260808 02:24:45.336251 25504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:45.337220 25504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:45.338478 25504 hybrid_clock.cc:648] HybridClock initialized: now 1786155885338421 us; error 41 us; skew 500 ppm
I20260808 02:24:45.338850 25504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:45.341181 25504 webserver.cc:460] Webserver started at http://127.19.231.2:35165/ using document root <none> and password file <none>
I20260808 02:24:45.341885 25504 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:45.342020 25504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:45.342383 25504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:45.345309 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f"
format_stamp: "Formatted at 2026-08-08 02:24:45 on dist-test-slave-pgkr"
server_key: "040de1a48986a59e8c533ead49766a02"
server_key_iv: "6035cbe786ab78dc09ab7d5bdfa7637c"
server_key_version: "encryptionkey@0"
I20260808 02:24:45.346119 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f"
format_stamp: "Formatted at 2026-08-08 02:24:45 on dist-test-slave-pgkr"
server_key: "040de1a48986a59e8c533ead49766a02"
server_key_iv: "6035cbe786ab78dc09ab7d5bdfa7637c"
server_key_version: "encryptionkey@0"
I20260808 02:24:45.351305 25504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:24:45.354789 25518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:45.356034 25504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:45.356220 25504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f"
format_stamp: "Formatted at 2026-08-08 02:24:45 on dist-test-slave-pgkr"
server_key: "040de1a48986a59e8c533ead49766a02"
server_key_iv: "6035cbe786ab78dc09ab7d5bdfa7637c"
server_key_version: "encryptionkey@0"
I20260808 02:24:45.356451 25504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:45.362176 25504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:45.362972 25504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:45.363209 25504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:45.364061 25504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:45.365484 25504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:45.365571 25504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:45.365654 25504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:45.365715 25504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:45.400895 25504 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:40499
I20260808 02:24:45.400975 25634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:40499 every 8 connection(s)
I20260808 02:24:45.402299 25504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:45.403694 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 25504
I20260808 02:24:45.403891 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:45.404409 20380 external_mini_cluster.cc:1428] Setting key 2e27cb8ea3ac8fb4a6791487635c4028
I20260808 02:24:45.407594 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35753
--builtin_ntp_servers=127.19.231.20:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 02:24:45.411628 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.405262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:41257 (local address 127.19.231.62:35753)
0808 02:24:45.405483 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:45.405489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:45.407364 (+ 1875us) server_negotiation.cc:407] Connection header received
0808 02:24:45.409200 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:45.409206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:45.409290 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:45.409419 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:45.410837 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:45.410843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:45.410848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:45.410869 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:45.410914 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:45.410926 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:45.411057 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:45.411254 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 02:24:45.411330 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:24:45.412798 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.405878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:35753 (local address 127.19.231.2:41257)
0808 02:24:45.407205 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:45.407245 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:45.408924 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:45.409657 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:45.409665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:45.409760 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:45.410483 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:45.410486 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:45.410525 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:45.410665 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:45.411024 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:45.411035 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:45.411049 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:24:45.411339 (+ 290us) client_negotiation.cc:240] Negotiation successful
0808 02:24:45.411603 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":586,"threads_started":1}
I20260808 02:24:45.414256 25635 heartbeater.cc:347] Connected to a master server at 127.19.231.62:35753
I20260808 02:24:45.414593 25635 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:45.415391 25635 heartbeater.cc:511] Master 127.19.231.62:35753 requested a full tablet report, sending...
I20260808 02:24:45.417095 25305 ts_manager.cc:194] Registered new tserver with Master: ac759a7b8622456cbd4b95cb1e5d0b1f (127.19.231.2:40499)
W20260808 02:24:45.758980 25639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:45.759456 25639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:45.759647 25639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:45.769445 25639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:45.769596 25639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:45.769795 25639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:45.779561 25639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:45.780938 25639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:45.782897 25639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:45.796212 25645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:45.796142 25646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:45.796180 25648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:45.797374 25639 server_base.cc:1034] running on GCE node
I20260808 02:24:45.798014 25639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:45.799173 25639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:45.800414 25639 hybrid_clock.cc:648] HybridClock initialized: now 1786155885800364 us; error 61 us; skew 500 ppm
I20260808 02:24:45.800763 25639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:45.803241 25639 webserver.cc:460] Webserver started at http://127.19.231.3:45957/ using document root <none> and password file <none>
I20260808 02:24:45.803968 25639 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:45.804105 25639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:45.804459 25639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:45.807564 25639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "c4237cca8b714380aa1345b22f926be4"
format_stamp: "Formatted at 2026-08-08 02:24:45 on dist-test-slave-pgkr"
server_key: "a2d9247154f8889316c61e8e08f4819c"
server_key_iv: "4e1d56607c4b46dc4238f68cf30cac53"
server_key_version: "encryptionkey@0"
I20260808 02:24:45.808393 25639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "c4237cca8b714380aa1345b22f926be4"
format_stamp: "Formatted at 2026-08-08 02:24:45 on dist-test-slave-pgkr"
server_key: "a2d9247154f8889316c61e8e08f4819c"
server_key_iv: "4e1d56607c4b46dc4238f68cf30cac53"
server_key_version: "encryptionkey@0"
I20260808 02:24:45.813849 25639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:24:45.817471 25654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:45.818877 25639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:45.819053 25639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "c4237cca8b714380aa1345b22f926be4"
format_stamp: "Formatted at 2026-08-08 02:24:45 on dist-test-slave-pgkr"
server_key: "a2d9247154f8889316c61e8e08f4819c"
server_key_iv: "4e1d56607c4b46dc4238f68cf30cac53"
server_key_version: "encryptionkey@0"
I20260808 02:24:45.819295 25639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:45.825140 25639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:45.825974 25639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:45.826270 25639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:45.827292 25639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:45.828768 25639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:45.828902 25639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:45.829039 25639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:45.829129 25639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:45.863790 25639 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:34137
I20260808 02:24:45.863850 25771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:34137 every 8 connection(s)
I20260808 02:24:45.865150 25639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:45.873147 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 25639
I20260808 02:24:45.873349 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:45.873430 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.867805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:48015 (local address 127.19.231.62:35753)
0808 02:24:45.867990 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:45.867995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:45.869384 (+ 1389us) server_negotiation.cc:407] Connection header received
0808 02:24:45.871210 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:45.871216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:45.871290 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:45.871404 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:45.872662 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:45.872668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:45.872673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:45.872699 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:45.872733 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:45.872744 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:45.872872 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:45.873068 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 02:24:45.873142 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:24:45.873934 20380 external_mini_cluster.cc:1428] Setting key 88f30e5b7ed2a2b93cec34a422deabb6
I20260808 02:24:45.874459 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.868362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:35753 (local address 127.19.231.3:48015)
0808 02:24:45.869232 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:45.869268 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:45.870929 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:45.871632 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:45.871640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:45.871735 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:45.872367 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:45.872370 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:45.872407 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:45.872490 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:45.872843 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:45.872854 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:45.872861 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:24:45.873157 (+ 296us) client_negotiation.cc:240] Negotiation successful
0808 02:24:45.873407 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":266,"threads_started":1}
I20260808 02:24:45.875909 25772 heartbeater.cc:347] Connected to a master server at 127.19.231.62:35753
I20260808 02:24:45.876302 25772 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:45.876995 25772 heartbeater.cc:511] Master 127.19.231.62:35753 requested a full tablet report, sending...
I20260808 02:24:45.878540 25304 ts_manager.cc:194] Registered new tserver with Master: c4237cca8b714380aa1345b22f926be4 (127.19.231.3:34137)
I20260808 02:24:45.889549 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:24:45.896931 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.894054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.19.231.62:35753)
0808 02:24:45.894225 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:45.894229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:45.894832 (+ 603us) server_negotiation.cc:407] Connection header received
0808 02:24:45.894957 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:45.894961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:45.895015 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:45.895478 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:45.896415 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:45.896419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:45.896422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:45.896438 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:45.896471 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:45.896482 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:45.896626 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:45.896669 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 02:24:45.896716 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:24:45.901108 25304 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:24:45.909327 25304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:24:45.932896 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.922646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:34137 (local address 127.0.0.1:48998)
0808 02:24:45.927609 (+ 4963us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:45.927651 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:45.927873 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:45.929947 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:45.929958 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:45.930116 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:45.931134 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:45.931139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:45.931186 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:45.931296 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:45.931972 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:45.931987 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:45.931999 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:24:45.932232 (+ 233us) client_negotiation.cc:240] Negotiation successful
0808 02:24:45.932359 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4712,"thread_start_us":5330,"threads_started":1}
I20260808 02:24:45.933218 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.923207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.19.231.3:34137)
0808 02:24:45.924178 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:45.924191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:45.927755 (+ 3564us) server_negotiation.cc:407] Connection header received
0808 02:24:45.928112 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:45.928125 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:45.928378 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:45.928526 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:45.931543 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:45.931557 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:45.931578 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:45.931602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:45.931671 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:45.931699 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:45.931904 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:45.932702 (+ 798us) server_negotiation.cc:299] Negotiation successful
0808 02:24:45.932815 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":546,"threads_started":1}
I20260808 02:24:45.935868 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.922707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47576)
0808 02:24:45.928263 (+ 5556us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:45.928284 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:45.928368 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:45.933819 (+ 5451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:45.933827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:45.933855 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:45.934627 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:45.934634 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:45.934653 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:45.934696 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:45.935218 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:45.935223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:45.935227 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:24:45.935386 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 02:24:45.935443 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5425,"thread_start_us":4504,"threads_started":1}
I20260808 02:24:45.936209 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.928754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:40499 (local address 127.0.0.1:39936)
0808 02:24:45.933195 (+ 4441us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:45.933216 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:45.933310 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:45.934269 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:45.934274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:45.934311 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:45.935124 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:45.935129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:45.935147 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:45.935193 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:45.935765 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:45.935779 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:45.935784 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 02:24:45.935898 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 02:24:45.935945 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4303,"thread_start_us":2232,"threads_started":1}
I20260808 02:24:45.936357 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.922708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.19.231.1:32835)
0808 02:24:45.932659 (+ 9951us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:45.932667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:45.932695 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:24:45.932763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:45.932771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:45.932983 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:45.933160 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:45.934874 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:45.934887 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:45.934899 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:45.934927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:45.934986 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:45.935007 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:45.935154 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:45.935911 (+ 757us) server_negotiation.cc:299] Negotiation successful
0808 02:24:45.936008 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9792,"thread_start_us":9667,"threads_started":1}
I20260808 02:24:45.936867 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:45.928790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.19.231.2:40499)
0808 02:24:45.933765 (+ 4975us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:45.933773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:45.933800 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:24:45.933865 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:45.933872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:45.934066 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:45.934663 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:45.935554 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:45.935562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:45.935574 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:45.935596 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:45.935636 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:45.935654 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:45.935796 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:45.936422 (+ 626us) server_negotiation.cc:299] Negotiation successful
0808 02:24:45.936526 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4796,"thread_start_us":4662,"threads_started":1}
I20260808 02:24:45.938225 25705 tablet_service.cc:1467] Processing CreateTablet for tablet ee15b045ffa14b05bcf25f2a9692f955 (DEFAULT_TABLE table=test-table [id=c1fafd74e2304d11ac20c78811ecfcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:45.939965 25705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee15b045ffa14b05bcf25f2a9692f955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:45.939950 25431 tablet_service.cc:1467] Processing CreateTablet for tablet ee15b045ffa14b05bcf25f2a9692f955 (DEFAULT_TABLE table=test-table [id=c1fafd74e2304d11ac20c78811ecfcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:45.941040 25569 tablet_service.cc:1467] Processing CreateTablet for tablet ee15b045ffa14b05bcf25f2a9692f955 (DEFAULT_TABLE table=test-table [id=c1fafd74e2304d11ac20c78811ecfcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:24:45.941679 25431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee15b045ffa14b05bcf25f2a9692f955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:45.942606 25569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee15b045ffa14b05bcf25f2a9692f955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:45.957504 25792 tablet_bootstrap.cc:492] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: Bootstrap starting.
I20260808 02:24:45.962476 25792 tablet_bootstrap.cc:654] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:45.963177 25791 tablet_bootstrap.cc:492] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Bootstrap starting.
I20260808 02:24:45.964135 25792 log.cc:824] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:45.965119 25500 heartbeater.cc:503] Master 127.19.231.62:35753 was elected leader, sending a full tablet report...
I20260808 02:24:45.966923 25791 tablet_bootstrap.cc:654] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:45.966915 25793 tablet_bootstrap.cc:492] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Bootstrap starting.
I20260808 02:24:45.968351 25791 log.cc:824] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:45.969619 25792 tablet_bootstrap.cc:492] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: No bootstrap required, opened a new log
I20260808 02:24:45.969902 25792 ts_tablet_manager.cc:1397] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260808 02:24:45.971376 25793 tablet_bootstrap.cc:654] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:45.972903 25793 log.cc:824] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:45.977371 25791 tablet_bootstrap.cc:492] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: No bootstrap required, opened a new log
I20260808 02:24:45.977684 25791 ts_tablet_manager.cc:1397] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260808 02:24:45.978205 25793 tablet_bootstrap.cc:492] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: No bootstrap required, opened a new log
I20260808 02:24:45.978489 25793 ts_tablet_manager.cc:1397] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260808 02:24:45.978499 25792 raft_consensus.cc:348] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:45.979012 25792 raft_consensus.cc:374] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:45.979135 25792 raft_consensus.cc:729] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec00b588f884a8e895617c476929794, State: Initialized, Role: FOLLOWER
I20260808 02:24:45.979766 25792 consensus_queue.cc:260] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:45.985291 25792 ts_tablet_manager.cc:1428] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260808 02:24:45.988366 25791 raft_consensus.cc:348] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:45.988835 25791 raft_consensus.cc:374] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:45.988934 25791 raft_consensus.cc:729] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac759a7b8622456cbd4b95cb1e5d0b1f, State: Initialized, Role: FOLLOWER
I20260808 02:24:45.989585 25791 consensus_queue.cc:260] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:45.989615 25793 raft_consensus.cc:348] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:45.990090 25793 raft_consensus.cc:374] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:45.990305 25793 raft_consensus.cc:729] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4237cca8b714380aa1345b22f926be4, State: Initialized, Role: FOLLOWER
I20260808 02:24:45.990825 25635 heartbeater.cc:503] Master 127.19.231.62:35753 was elected leader, sending a full tablet report...
I20260808 02:24:45.991529 25791 ts_tablet_manager.cc:1428] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260808 02:24:45.992450 25793 consensus_queue.cc:260] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:45.993721 25772 heartbeater.cc:503] Master 127.19.231.62:35753 was elected leader, sending a full tablet report...
I20260808 02:24:45.994071 25793 ts_tablet_manager.cc:1428] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
W20260808 02:24:46.117002 25773 tablet.cc:2369] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:46.143126 25799 raft_consensus.cc:482] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:24:46.143468 25799 raft_consensus.cc:504] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:46.145197 25799 leader_election.cc:288] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec00b588f884a8e895617c476929794 (127.19.231.1:32835), c4237cca8b714380aa1345b22f926be4 (127.19.231.3:34137)
I20260808 02:24:46.149191 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.145775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:45069 (local address 127.19.231.1:32835)
0808 02:24:46.145957 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:46.145965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:46.146869 (+ 904us) server_negotiation.cc:407] Connection header received
0808 02:24:46.146961 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:46.146967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:46.147040 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:46.147167 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:46.148457 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:46.148464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:46.148469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:46.148488 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:46.148533 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:46.148551 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:46.148675 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:46.148800 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 02:24:46.148870 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:24:46.149191 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.145668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:54337 (local address 127.19.231.3:34137)
0808 02:24:46.145867 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:46.145873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:46.146836 (+ 963us) server_negotiation.cc:407] Connection header received
0808 02:24:46.146961 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:46.146967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:46.147046 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:46.147167 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:46.148435 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:46.148443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:46.148447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:46.148469 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:46.148514 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:46.148533 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:46.148681 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:46.148801 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 02:24:46.148870 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:24:46.149479 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.145518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:34137 (local address 127.19.231.2:54337)
0808 02:24:46.146679 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.146699 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:46.146848 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:46.147211 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:46.147216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:46.147242 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:46.148207 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:46.148209 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:46.148220 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:46.148246 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:46.148675 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:46.148679 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:46.148683 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:24:46.148836 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 02:24:46.148902 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":639,"threads_started":1}
I20260808 02:24:46.149501 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.145638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.19.231.2:45069)
0808 02:24:46.146754 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.146771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:46.146852 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:46.147211 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:46.147216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:46.147242 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:46.148147 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:24:46.148171 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:46.148193 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:24:46.148245 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:46.148653 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:46.148658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:46.148662 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 02:24:46.148873 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 02:24:46.148907 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":431,"threads_started":1}
I20260808 02:24:46.151003 25727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee15b045ffa14b05bcf25f2a9692f955" candidate_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4237cca8b714380aa1345b22f926be4" is_pre_election: true
I20260808 02:24:46.151003 25451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee15b045ffa14b05bcf25f2a9692f955" candidate_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec00b588f884a8e895617c476929794" is_pre_election: true
I20260808 02:24:46.151599 25451 raft_consensus.cc:2454] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac759a7b8622456cbd4b95cb1e5d0b1f in term 0.
I20260808 02:24:46.151599 25727 raft_consensus.cc:2454] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac759a7b8622456cbd4b95cb1e5d0b1f in term 0.
I20260808 02:24:46.152400 25519 leader_election.cc:302] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ac759a7b8622456cbd4b95cb1e5d0b1f, c4237cca8b714380aa1345b22f926be4, fec00b588f884a8e895617c476929794; no voters:
I20260808 02:24:46.152839 25799 raft_consensus.cc:2781] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:24:46.152990 25799 raft_consensus.cc:482] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:24:46.153115 25799 raft_consensus.cc:3037] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 0 FOLLOWER]: Advancing to term 1
W20260808 02:24:46.154356 25637 tablet.cc:2369] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:46.155612 25799 raft_consensus.cc:504] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:46.156219 25799 leader_election.cc:288] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [CANDIDATE]: Term 1 election: Requested vote from peers fec00b588f884a8e895617c476929794 (127.19.231.1:32835), c4237cca8b714380aa1345b22f926be4 (127.19.231.3:34137)
I20260808 02:24:46.156735 25451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee15b045ffa14b05bcf25f2a9692f955" candidate_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec00b588f884a8e895617c476929794"
I20260808 02:24:46.156786 25727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee15b045ffa14b05bcf25f2a9692f955" candidate_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4237cca8b714380aa1345b22f926be4"
I20260808 02:24:46.156944 25451 raft_consensus.cc:3037] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:46.156982 25727 raft_consensus.cc:3037] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:46.159231 25727 raft_consensus.cc:2454] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac759a7b8622456cbd4b95cb1e5d0b1f in term 1.
I20260808 02:24:46.159341 25451 raft_consensus.cc:2454] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac759a7b8622456cbd4b95cb1e5d0b1f in term 1.
I20260808 02:24:46.159680 25522 leader_election.cc:302] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac759a7b8622456cbd4b95cb1e5d0b1f, c4237cca8b714380aa1345b22f926be4; no voters:
I20260808 02:24:46.160005 25799 raft_consensus.cc:2781] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:46.160457 25799 raft_consensus.cc:686] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 1 LEADER]: Becoming Leader. State: Replica: ac759a7b8622456cbd4b95cb1e5d0b1f, State: Running, Role: LEADER
I20260808 02:24:46.161010 25799 consensus_queue.cc:237] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } }
I20260808 02:24:46.165695 25305 catalog_manager.cc:5495] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac759a7b8622456cbd4b95cb1e5d0b1f (127.19.231.2). New cstate: current_term: 1 leader_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac759a7b8622456cbd4b95cb1e5d0b1f" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40499 } health_report { overall_health: HEALTHY } } }
I20260808 02:24:46.178819 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.176776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.19.231.2:40499)
0808 02:24:46.176931 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:46.176936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:46.176953 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:24:46.177142 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:46.177146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:46.177229 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:46.177315 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:46.178244 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:46.178247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:46.178250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:46.178263 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:46.178291 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:46.178302 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:46.178389 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:46.178483 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:24:46.178529 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 02:24:46.186951 25727 raft_consensus.cc:1261] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 1 FOLLOWER]: Refusing update from remote peer ac759a7b8622456cbd4b95cb1e5d0b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:46.187856 25451 raft_consensus.cc:1261] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 1 FOLLOWER]: Refusing update from remote peer ac759a7b8622456cbd4b95cb1e5d0b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:24:46.188226 25803 consensus_queue.cc:1035] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4237cca8b714380aa1345b22f926be4" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:24:46.188755 25799 consensus_queue.cc:1035] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec00b588f884a8e895617c476929794" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:24:46.199802 25501 tablet.cc:2369] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:24:46.201344 25806 mvcc.cc:205] Tried to move back new op lower bound from 7316094509811539968 to 7316094509719588864. Current Snapshot: MvccSnapshot[applied={T|T < 7316094509811539968}]
I20260808 02:24:46.203583 25808 mvcc.cc:205] Tried to move back new op lower bound from 7316094509811539968 to 7316094509719588864. Current Snapshot: MvccSnapshot[applied={T|T < 7316094509811539968}]
I20260808 02:24:46.215734 25304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:46.216090 25304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:24:46.220243 25304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 728cbb8ec25c44d19210748db4fafae8: Sending DeleteTablet for 3 replicas of tablet ee15b045ffa14b05bcf25f2a9692f955
I20260808 02:24:46.221714 25569 tablet_service.cc:1514] Processing DeleteTablet for tablet ee15b045ffa14b05bcf25f2a9692f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:46 UTC) from {username='slave'} at 127.0.0.1:39936
I20260808 02:24:46.221846 25431 tablet_service.cc:1514] Processing DeleteTablet for tablet ee15b045ffa14b05bcf25f2a9692f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:46 UTC) from {username='slave'} at 127.0.0.1:47576
I20260808 02:24:46.221846 25705 tablet_service.cc:1514] Processing DeleteTablet for tablet ee15b045ffa14b05bcf25f2a9692f955 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:24:46 UTC) from {username='slave'} at 127.0.0.1:48998
I20260808 02:24:46.224684 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 25366
I20260808 02:24:46.232687 25819 tablet_replica.cc:330] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: stopping tablet replica
I20260808 02:24:46.234117 25819 raft_consensus.cc:2229] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:46.236469 25819 raft_consensus.cc:2258] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:46.239801 25819 ts_tablet_manager.cc:1905] T ee15b045ffa14b05bcf25f2a9692f955 P fec00b588f884a8e895617c476929794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:46.242282 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.226696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.19.231.1:32835)
0808 02:24:46.226946 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:46.226956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:46.238989 (+ 12033us) server_negotiation.cc:407] Connection header received
0808 02:24:46.239579 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:46.239585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:46.239680 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:46.239765 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:46.241388 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:46.241399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:46.241408 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:46.241436 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:46.241489 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:46.241513 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:46.241640 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:46.241902 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 02:24:46.241968 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 02:24:46.243681 25495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:46.244019 25817 tablet_replica.cc:330] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: stopping tablet replica
I20260808 02:24:46.244496 25818 tablet_replica.cc:330] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: stopping tablet replica
I20260808 02:24:46.244580 25817 raft_consensus.cc:2229] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:24:46.244956 25818 raft_consensus.cc:2229] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:24:46.245291 25817 raft_consensus.cc:2258] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:46.245496 25818 raft_consensus.cc:2258] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:24:46.247516 25817 ts_tablet_manager.cc:1905] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:46.248454 25818 ts_tablet_manager.cc:1905] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:24:46.256691 25817 ts_tablet_manager.cc:1918] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:46.257063 25817 log.cc:1196] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/ee15b045ffa14b05bcf25f2a9692f955
I20260808 02:24:46.257858 25817 ts_tablet_manager.cc:1939] T ee15b045ffa14b05bcf25f2a9692f955 P ac759a7b8622456cbd4b95cb1e5d0b1f: Deleting consensus metadata
I20260808 02:24:46.257905 25818 ts_tablet_manager.cc:1918] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:24:46.258164 25818 log.cc:1196] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/ee15b045ffa14b05bcf25f2a9692f955
I20260808 02:24:46.258744 25818 ts_tablet_manager.cc:1939] T ee15b045ffa14b05bcf25f2a9692f955 P c4237cca8b714380aa1345b22f926be4: Deleting consensus metadata
I20260808 02:24:46.260141 25291 catalog_manager.cc:4841] TS ac759a7b8622456cbd4b95cb1e5d0b1f (127.19.231.2:40499): tablet ee15b045ffa14b05bcf25f2a9692f955 (table test-table [id=c1fafd74e2304d11ac20c78811ecfcd7]) successfully deleted
I20260808 02:24:46.260560 25291 catalog_manager.cc:4841] TS c4237cca8b714380aa1345b22f926be4 (127.19.231.3:34137): tablet ee15b045ffa14b05bcf25f2a9692f955 (table test-table [id=c1fafd74e2304d11ac20c78811ecfcd7]) successfully deleted
I20260808 02:24:46.345831 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 25366
I20260808 02:24:46.375213 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 25504
W20260808 02:24:46.375170 25289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.231.1:32835 (error 108)
I20260808 02:24:46.379132 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.376384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.19.231.2:40499)
0808 02:24:46.376620 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:46.376628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:46.376667 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:24:46.376779 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:46.376788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:46.376874 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:46.377007 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:46.378068 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:46.378074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:46.378079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:46.378100 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:46.378157 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:46.378180 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:46.378311 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:46.378738 (+ 427us) server_negotiation.cc:299] Negotiation successful
0808 02:24:46.378798 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:24:46.380661 25630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:46.387362 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.386915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47596)
0808 02:24:46.387084 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.387223 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 02:24:46.387650 25289 catalog_manager.cc:4584] TS fec00b588f884a8e895617c476929794 (127.19.231.1:32835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee15b045ffa14b05bcf25f2a9692f955: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
I20260808 02:24:46.441193 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.440721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47610)
0808 02:24:46.440935 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.441030 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 02:24:46.442204 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.441856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47622)
0808 02:24:46.442008 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.442095 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 02:24:46.495036 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 25504
I20260808 02:24:46.502964 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.502325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47630)
0808 02:24:46.502599 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.502764 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 02:24:46.504156 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.503702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47646)
0808 02:24:46.503878 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.503970 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 02:24:46.520474 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 25639
I20260808 02:24:46.523641 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.521325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.19.231.3:34137)
0808 02:24:46.521671 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:46.521677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:46.521701 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:24:46.521776 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:46.521781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:46.521882 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:46.521997 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:46.522949 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:46.522955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:46.522960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:46.522979 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:46.523019 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:46.523032 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:46.523182 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:46.523242 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 02:24:46.523304 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260808 02:24:46.524241 25767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:46.585011 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.584481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47652)
0808 02:24:46.584756 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.584855 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260808 02:24:46.586228 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:46.585760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:32835 (local address 127.0.0.1:47668)
0808 02:24:46.585996 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:46.586068 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260808 02:24:46.614674 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 25639
I20260808 02:24:46.635479 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 25273
I20260808 02:24:46.637128 25335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:46.744832 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 25273
2026-08-08T02:24:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2878 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 02:24:46.792716 20380 mini_oidc.cc:152] Starting JWKS server
I20260808 02:24:46.795480 20380 webserver.cc:460] Webserver started at https://127.0.0.1:43183/ using document root <none> and password file <none>
I20260808 02:24:46.795746 20380 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:24:46.796342 20380 webserver.cc:460] Webserver started at http://127.0.0.1:36821/ using document root <none> and password file <none>
2026-08-08T02:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:46Z Disabled control of system clock
I20260808 02:24:46.831903 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44207
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:44207
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/testchainca.cert with env {}
W20260808 02:24:47.166745 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:47.167199 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:47.167387 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:47.174845 25831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:47.174942 25831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:47.174981 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:47.175037 25831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:47.175086 25831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:24:47.175141 25831 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:24:47.175174 25831 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43183/jwks.json
I20260808 02:24:47.185122 25831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:44207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:47.186359 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:47.187640 25831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:47.199788 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:47.199848 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:47.199870 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:47.201042 25831 server_base.cc:1034] running on GCE node
I20260808 02:24:47.201924 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:47.203497 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:47.204794 25831 hybrid_clock.cc:648] HybridClock initialized: now 1786155887204744 us; error 37 us; skew 500 ppm
I20260808 02:24:47.205168 25831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:47.212328 25831 webserver.cc:460] Webserver started at http://127.19.231.62:40443/ using document root <none> and password file <none>
I20260808 02:24:47.213187 25831 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:47.213328 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:47.213677 25831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:47.216779 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "b56e1faa9f4248b48d6903beaee59b7e"
format_stamp: "Formatted at 2026-08-08 02:24:47 on dist-test-slave-pgkr"
server_key: "a0d2808f593797ee117fdfb8baf6a9c9"
server_key_iv: "85634a8dd62956bf8044991a8869479e"
server_key_version: "encryptionkey@0"
I20260808 02:24:47.217605 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "b56e1faa9f4248b48d6903beaee59b7e"
format_stamp: "Formatted at 2026-08-08 02:24:47 on dist-test-slave-pgkr"
server_key: "a0d2808f593797ee117fdfb8baf6a9c9"
server_key_iv: "85634a8dd62956bf8044991a8869479e"
server_key_version: "encryptionkey@0"
I20260808 02:24:47.222980 25831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:24:47.226553 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:47.227944 25831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:47.228140 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "b56e1faa9f4248b48d6903beaee59b7e"
format_stamp: "Formatted at 2026-08-08 02:24:47 on dist-test-slave-pgkr"
server_key: "a0d2808f593797ee117fdfb8baf6a9c9"
server_key_iv: "85634a8dd62956bf8044991a8869479e"
server_key_version: "encryptionkey@0"
I20260808 02:24:47.228360 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:47.243464 25831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:47.244278 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:47.244691 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:47.267536 25831 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:44207
I20260808 02:24:47.267562 25901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:44207 every 8 connection(s)
I20260808 02:24:47.268929 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:47.273290 25902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:47.274384 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 25831
I20260808 02:24:47.274696 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:47.275746 20380 external_mini_cluster.cc:1428] Setting key 8af8aaa5731dbdc43b55f59290dc83e3
I20260808 02:24:47.288416 25902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e: Bootstrap starting.
I20260808 02:24:47.289747 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:47.277968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.19.231.62:44207)
0808 02:24:47.279108 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:47.279120 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:47.279162 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:24:47.280950 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:47.280972 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:47.281419 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:47.281842 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:47.282392 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:47.283712 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:47.285065 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:47.285802 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:47.286029 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:47.287408 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:47.287415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:47.287428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:47.288005 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:47.288093 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:47.288115 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:47.288250 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:47.288487 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 02:24:47.288826 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":402,"threads_started":1}
I20260808 02:24:47.292836 25902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:47.294083 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:24:47.294275 25902 log.cc:824] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:47.297717 25902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e: No bootstrap required, opened a new log
I20260808 02:24:47.306389 25902 raft_consensus.cc:348] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e1faa9f4248b48d6903beaee59b7e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44207 } }
I20260808 02:24:47.306933 25902 raft_consensus.cc:374] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:47.307070 25902 raft_consensus.cc:729] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56e1faa9f4248b48d6903beaee59b7e, State: Initialized, Role: FOLLOWER
I20260808 02:24:47.307732 25902 consensus_queue.cc:260] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e1faa9f4248b48d6903beaee59b7e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44207 } }
I20260808 02:24:47.308012 25902 raft_consensus.cc:388] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:47.308194 25902 raft_consensus.cc:482] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:47.308404 25902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:47.311283 25902 raft_consensus.cc:504] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e1faa9f4248b48d6903beaee59b7e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44207 } }
I20260808 02:24:47.311896 25902 leader_election.cc:302] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b56e1faa9f4248b48d6903beaee59b7e; no voters:
I20260808 02:24:47.312587 25902 leader_election.cc:288] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:47.312686 25908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:47.313586 25908 raft_consensus.cc:686] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [term 1 LEADER]: Becoming Leader. State: Replica: b56e1faa9f4248b48d6903beaee59b7e, State: Running, Role: LEADER
I20260808 02:24:47.314182 25908 consensus_queue.cc:237] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e1faa9f4248b48d6903beaee59b7e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44207 } }
I20260808 02:24:47.315264 25902 sys_catalog.cc:564] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:47.317850 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b56e1faa9f4248b48d6903beaee59b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e1faa9f4248b48d6903beaee59b7e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44207 } } }
I20260808 02:24:47.318245 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:47.317421 25910 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b56e1faa9f4248b48d6903beaee59b7e. Latest consensus state: current_term: 1 leader_uuid: "b56e1faa9f4248b48d6903beaee59b7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56e1faa9f4248b48d6903beaee59b7e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44207 } } }
I20260808 02:24:47.318953 25910 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:47.323009 25914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:47.328879 25914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:47.338325 25914 catalog_manager.cc:1347] Generated new cluster ID: 723085b27ab3461ba7c97f44b70777c6
I20260808 02:24:47.338485 25914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:47.361357 25914 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:47.362769 25914 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:47.374140 25914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b56e1faa9f4248b48d6903beaee59b7e: Generated new TSK 0
I20260808 02:24:47.375073 25914 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:24:47.410385 25934 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 02:24:47.410573 25934 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:24:47.411886 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:47.389616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.19.231.62:44207)
0808 02:24:47.389810 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:47.389817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:47.390308 (+ 491us) server_negotiation.cc:407] Connection header received
0808 02:24:47.390703 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:47.390714 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:47.390802 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:47.390979 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:24:47.391506 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:47.392660 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:47.394056 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:47.394510 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:47.394673 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:47.395235 (+ 562us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:24:47.411183 (+ 15948us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:24:47.411552 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 02:24:47.412024 25905 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 02:24:47.413368 25933 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:47.389517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:44207 (local address 127.0.0.1:36880)
0808 02:24:47.390211 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:47.390228 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:47.390370 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:47.391016 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:47.391020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:47.391030 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:24:47.391361 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:47.391368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:47.392794 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:47.392798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:47.393906 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:47.393916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:47.394707 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:47.394713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:47.394864 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:47.394918 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:24:47.411466 (+ 16548us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 02:24:47.411745 (+ 279us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.231.62:44207: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":352,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.231.62:44207: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 02:24:47.415052 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 25831
I20260808 02:24:47.416318 25897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:47.512676 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 25831
2026-08-08T02:24:47Z chronyd exiting
I20260808 02:24:47.567281 20380 test_util.cc:175] -----------------------------------------------
I20260808 02:24:47.567430 20380 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786155839876607-20380-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (779 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 02:24:47.571161 20380 mini_oidc.cc:152] Starting JWKS server
I20260808 02:24:47.572969 20380 webserver.cc:460] Webserver started at https://127.0.0.1:44335/ using document root <none> and password file <none>
I20260808 02:24:47.573143 20380 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:24:47.573752 20380 webserver.cc:460] Webserver started at http://127.0.0.1:41105/ using document root <none> and password file <none>
2026-08-08T02:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:47Z Disabled control of system clock
I20260808 02:24:47.608307 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:33649
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:43009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:33649
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44335/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/testchainca.cert with env {}
W20260808 02:24:47.953106 25941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:47.953397 25941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:47.953608 25941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:47.960781 25941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:47.960916 25941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:47.960989 25941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:47.961089 25941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:47.961180 25941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:24:47.961272 25941 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:24:47.961331 25941 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44335/jwks.json
I20260808 02:24:47.971678 25941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:43009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:33649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:33649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44335/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:47.973075 25941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:47.974550 25941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:47.987006 25946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:47.987078 25947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:47.987042 25950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:47.987995 25941 server_base.cc:1034] running on GCE node
I20260808 02:24:47.988960 25941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:47.990450 25941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:47.991696 25941 hybrid_clock.cc:648] HybridClock initialized: now 1786155887991650 us; error 61 us; skew 500 ppm
I20260808 02:24:47.992069 25941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:47.994520 25941 webserver.cc:460] Webserver started at http://127.19.231.62:37377/ using document root <none> and password file <none>
I20260808 02:24:47.995296 25941 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:47.995445 25941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:47.995800 25941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:47.999006 25941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "a5ed598a3bff48dbb411be6431fbe9c1"
format_stamp: "Formatted at 2026-08-08 02:24:47 on dist-test-slave-pgkr"
server_key: "a6a246c3b4769d2fe904098e75af4ed1"
server_key_iv: "8508ea0ec1adf974d1133038092723d4"
server_key_version: "encryptionkey@0"
I20260808 02:24:47.999832 25941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "a5ed598a3bff48dbb411be6431fbe9c1"
format_stamp: "Formatted at 2026-08-08 02:24:47 on dist-test-slave-pgkr"
server_key: "a6a246c3b4769d2fe904098e75af4ed1"
server_key_iv: "8508ea0ec1adf974d1133038092723d4"
server_key_version: "encryptionkey@0"
I20260808 02:24:48.005191 25941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:24:48.008864 25956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:48.010341 25941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:24:48.010542 25941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "a5ed598a3bff48dbb411be6431fbe9c1"
format_stamp: "Formatted at 2026-08-08 02:24:47 on dist-test-slave-pgkr"
server_key: "a6a246c3b4769d2fe904098e75af4ed1"
server_key_iv: "8508ea0ec1adf974d1133038092723d4"
server_key_version: "encryptionkey@0"
I20260808 02:24:48.010855 25941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:48.024473 25941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:48.025326 25941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:48.025665 25941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:48.045034 25941 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:33649
I20260808 02:24:48.045038 26011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:33649 every 8 connection(s)
I20260808 02:24:48.046453 25941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:48.050223 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 25941
I20260808 02:24:48.050416 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:48.050629 26012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:48.051321 20380 external_mini_cluster.cc:1428] Setting key 8c886ce99e5cb705c32e23a45f8564fb
I20260808 02:24:48.064257 26012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1: Bootstrap starting.
I20260808 02:24:48.065061 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:48.053335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.19.231.62:33649)
0808 02:24:48.054422 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:48.054434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:48.054475 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:24:48.056081 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:48.056103 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:48.056589 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:48.057005 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:48.057600 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.058964 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.060234 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.060957 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.061209 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:48.062586 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:48.062602 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:48.062663 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:48.063252 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:48.063330 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:48.063351 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:48.063490 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:48.063677 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 02:24:48.063992 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":384,"threads_started":1}
I20260808 02:24:48.067734 26012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:48.068955 26012 log.cc:824] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:48.069190 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:24:48.072157 26012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1: No bootstrap required, opened a new log
I20260808 02:24:48.080684 26012 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33649 } }
I20260808 02:24:48.081066 26012 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:48.081195 26012 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ed598a3bff48dbb411be6431fbe9c1, State: Initialized, Role: FOLLOWER
I20260808 02:24:48.081844 26012 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33649 } }
I20260808 02:24:48.082060 26012 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:48.082221 26012 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:48.082394 26012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:48.084901 26012 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33649 } }
I20260808 02:24:48.085489 26012 leader_election.cc:302] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5ed598a3bff48dbb411be6431fbe9c1; no voters:
I20260808 02:24:48.086174 26012 leader_election.cc:288] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:48.086330 26017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:48.087154 26017 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [term 1 LEADER]: Becoming Leader. State: Replica: a5ed598a3bff48dbb411be6431fbe9c1, State: Running, Role: LEADER
I20260808 02:24:48.087747 26017 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33649 } }
I20260808 02:24:48.088446 26012 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:48.091015 26019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ed598a3bff48dbb411be6431fbe9c1. Latest consensus state: current_term: 1 leader_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33649 } } }
I20260808 02:24:48.091470 26019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:48.091027 26018 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ed598a3bff48dbb411be6431fbe9c1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33649 } } }
I20260808 02:24:48.095867 26018 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:48.103826 26024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:48.109251 26024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:48.118160 26024 catalog_manager.cc:1347] Generated new cluster ID: 32c6242782e3448c8b5daa1bebf9a9cc
I20260808 02:24:48.118302 26024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:48.139249 26024 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:48.140736 26024 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:48.148707 26024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5ed598a3bff48dbb411be6431fbe9c1: Generated new TSK 0
I20260808 02:24:48.149536 26024 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 02:24:48.190516 26043 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 02:24:48.190697 26043 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:24:48.191764 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:48.163363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.19.231.62:33649)
0808 02:24:48.163600 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:48.163613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:48.164160 (+ 547us) server_negotiation.cc:407] Connection header received
0808 02:24:48.164406 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:48.164411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:48.164480 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:48.164606 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:24:48.165175 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.165916 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.168087 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.168565 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.168676 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:48.169338 (+ 662us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:24:48.191160 (+ 21822us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:24:48.191513 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 02:24:48.191916 26015 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 02:24:48.192168 26042 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:48.163349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:33649 (local address 127.0.0.1:56872)
0808 02:24:48.164043 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:48.164068 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:48.164222 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:48.164671 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:48.164677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:48.164690 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:24:48.165004 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:48.165016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.166070 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.166077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:48.167903 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:48.167920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.168770 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.168777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:48.168985 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:48.169034 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:24:48.191428 (+ 22394us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 02:24:48.191744 (+ 316us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.231.62:33649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":379,"threads_started":1}
I20260808 02:24:48.193761 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 25941
I20260808 02:24:48.195063 26007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:48.290563 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 25941
2026-08-08T02:24:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (779 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 02:24:48.351210 20380 mini_oidc.cc:152] Starting JWKS server
I20260808 02:24:48.353173 20380 webserver.cc:460] Webserver started at https://127.0.0.1:46575/ using document root <none> and password file <none>
I20260808 02:24:48.353434 20380 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 02:24:48.354074 20380 webserver.cc:460] Webserver started at http://127.0.0.1:44845/ using document root <none> and password file <none>
2026-08-08T02:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:48Z Disabled control of system clock
I20260808 02:24:48.389712 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40097
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:40097
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46575/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:24:48.724238 26050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:48.724514 26050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:48.724690 26050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:48.731822 26050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:48.731926 26050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:48.732020 26050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:48.732132 26050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:48.732213 26050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:24:48.732342 26050 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:24:48.732419 26050 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46575/jwks.json
I20260808 02:24:48.741653 26050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:40097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46575/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:48.743103 26050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:48.744930 26050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:48.756919 26058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:48.756959 26056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:48.756997 26055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:48.757908 26050 server_base.cc:1034] running on GCE node
I20260808 02:24:48.758742 26050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:48.760161 26050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:48.761401 26050 hybrid_clock.cc:648] HybridClock initialized: now 1786155888761363 us; error 48 us; skew 500 ppm
I20260808 02:24:48.761780 26050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:48.764217 26050 webserver.cc:460] Webserver started at http://127.19.231.62:42579/ using document root <none> and password file <none>
I20260808 02:24:48.765159 26050 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:48.765288 26050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:48.765700 26050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:48.769493 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "99047c44f7cc4d8c97228a8474839dbe"
format_stamp: "Formatted at 2026-08-08 02:24:48 on dist-test-slave-pgkr"
server_key: "c538eda96b7ebc4423cfffd0765c3b8a"
server_key_iv: "e3025484e7dd22f11b86afb082050318"
server_key_version: "encryptionkey@0"
I20260808 02:24:48.770335 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "99047c44f7cc4d8c97228a8474839dbe"
format_stamp: "Formatted at 2026-08-08 02:24:48 on dist-test-slave-pgkr"
server_key: "c538eda96b7ebc4423cfffd0765c3b8a"
server_key_iv: "e3025484e7dd22f11b86afb082050318"
server_key_version: "encryptionkey@0"
I20260808 02:24:48.775561 26050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:24:48.778913 26064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:48.780287 26050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:24:48.780434 26050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "99047c44f7cc4d8c97228a8474839dbe"
format_stamp: "Formatted at 2026-08-08 02:24:48 on dist-test-slave-pgkr"
server_key: "c538eda96b7ebc4423cfffd0765c3b8a"
server_key_iv: "e3025484e7dd22f11b86afb082050318"
server_key_version: "encryptionkey@0"
I20260808 02:24:48.780628 26050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:48.802378 26050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:48.803256 26050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:48.803541 26050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:48.823640 26050 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:40097
I20260808 02:24:48.823674 26118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:40097 every 8 connection(s)
I20260808 02:24:48.825103 26050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:48.828922 26119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:48.831611 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26050
I20260808 02:24:48.831838 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:48.832654 20380 external_mini_cluster.cc:1428] Setting key ef12c7834154966e09e5d5fa5c7611a0
I20260808 02:24:48.841703 26119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe: Bootstrap starting.
I20260808 02:24:48.844821 26119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:48.845901 26119 log.cc:824] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:48.848006 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:48.834832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.19.231.62:40097)
0808 02:24:48.835960 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:48.835973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:48.836016 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:24:48.838099 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:48.838129 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:48.838761 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:48.839243 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:48.839959 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.841551 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.843168 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.843936 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.844160 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:48.845435 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:48.845448 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:48.845468 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:48.846115 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:48.846223 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:48.846246 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:48.846383 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:48.846700 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 02:24:48.847025 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":358,"threads_started":1}
I20260808 02:24:48.848912 26119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe: No bootstrap required, opened a new log
I20260808 02:24:48.852062 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:24:48.856961 26119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99047c44f7cc4d8c97228a8474839dbe" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40097 } }
I20260808 02:24:48.857331 26119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:48.857440 26119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99047c44f7cc4d8c97228a8474839dbe, State: Initialized, Role: FOLLOWER
I20260808 02:24:48.857975 26119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99047c44f7cc4d8c97228a8474839dbe" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40097 } }
I20260808 02:24:48.858223 26119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:48.858369 26119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:48.858541 26119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:48.860793 26119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99047c44f7cc4d8c97228a8474839dbe" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40097 } }
I20260808 02:24:48.861392 26119 leader_election.cc:302] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99047c44f7cc4d8c97228a8474839dbe; no voters:
I20260808 02:24:48.862046 26119 leader_election.cc:288] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:48.862143 26125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:48.863058 26125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [term 1 LEADER]: Becoming Leader. State: Replica: 99047c44f7cc4d8c97228a8474839dbe, State: Running, Role: LEADER
I20260808 02:24:48.863669 26125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99047c44f7cc4d8c97228a8474839dbe" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40097 } }
I20260808 02:24:48.864323 26119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:48.866685 26127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99047c44f7cc4d8c97228a8474839dbe. Latest consensus state: current_term: 1 leader_uuid: "99047c44f7cc4d8c97228a8474839dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99047c44f7cc4d8c97228a8474839dbe" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40097 } } }
I20260808 02:24:48.866986 26126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99047c44f7cc4d8c97228a8474839dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99047c44f7cc4d8c97228a8474839dbe" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40097 } } }
I20260808 02:24:48.867390 26126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:48.867843 26127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe [sys.catalog]: This master's current role is: LEADER
W20260808 02:24:48.878453 26141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:24:48.878595 26141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:24:48.878824 26132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:48.880708 26132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:48.889590 26132 catalog_manager.cc:1347] Generated new cluster ID: 5790277576e54bd4b4c6dd53ae4693b7
I20260808 02:24:48.889757 26132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:48.905829 26132 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:48.907699 26132 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:48.918183 26132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99047c44f7cc4d8c97228a8474839dbe: Generated new TSK 0
I20260808 02:24:48.919293 26132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:24:48.942133 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:48.932860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.19.231.62:40097)
0808 02:24:48.933059 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:48.933065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:48.933477 (+ 412us) server_negotiation.cc:407] Connection header received
0808 02:24:48.933790 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:48.933798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:48.933901 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:48.934077 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:24:48.934533 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.935688 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.937250 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.937939 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.938079 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:48.938687 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:24:48.941123 (+ 2436us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:24:48.941801 (+ 678us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 02:24:48.942284 26123 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 02:24:48.943984 26149 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:48.932708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40097 (local address 127.0.0.1:55154)
0808 02:24:48.933362 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:48.933380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:48.933511 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:48.934104 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:48.934108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:48.934118 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:24:48.934381 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:48.934389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.935855 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.935859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:48.937096 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:48.937109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:48.938149 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:48.938154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:48.938311 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:48.938346 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:24:48.941455 (+ 3109us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 02:24:48.943496 (+ 2041us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.231.62:40097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":1709,"thread_start_us":373,"threads_started":1}
I20260808 02:24:48.945734 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26050
I20260808 02:24:48.946992 26114 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 02:24:48.957141 26150 webserver.cc:533] Webserver: sslize failed:
E20260808 02:24:48.957237 26150 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 02:24:49.042135 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26050
2026-08-08T02:24:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (752 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T02:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:49Z Disabled control of system clock
I20260808 02:24:49.139375 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37805
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37805
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/testchainca.cert with env {}
W20260808 02:24:49.480136 26155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:49.480630 26155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:49.480791 26155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:49.487993 26155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:49.488168 26155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:49.488243 26155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:49.488329 26155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:49.488412 26155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 02:24:49.488504 26155 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 02:24:49.488561 26155 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 02:24:49.498569 26155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:49.500041 26155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:49.501436 26155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:49.513741 26161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:49.514050 26164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:49.513717 26162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:49.515014 26155 server_base.cc:1034] running on GCE node
I20260808 02:24:49.515846 26155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:49.517318 26155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:49.518553 26155 hybrid_clock.cc:648] HybridClock initialized: now 1786155889518507 us; error 42 us; skew 500 ppm
I20260808 02:24:49.518942 26155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:24:49.521270 26155 webserver.cc:460] Webserver started at http://127.19.231.62:42819/ using document root <none> and password file <none>
I20260808 02:24:49.521996 26155 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:49.522133 26155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:49.522464 26155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:49.525449 26155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "9654461edfff4793bc8e6709609856db"
format_stamp: "Formatted at 2026-08-08 02:24:49 on dist-test-slave-pgkr"
server_key: "303cd59165d250158ada53ad576f6a5e"
server_key_iv: "a168d53e4a50eaa4587fbdacacbd9fa8"
server_key_version: "encryptionkey@0"
I20260808 02:24:49.526273 26155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "9654461edfff4793bc8e6709609856db"
format_stamp: "Formatted at 2026-08-08 02:24:49 on dist-test-slave-pgkr"
server_key: "303cd59165d250158ada53ad576f6a5e"
server_key_iv: "a168d53e4a50eaa4587fbdacacbd9fa8"
server_key_version: "encryptionkey@0"
I20260808 02:24:49.531574 26155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 02:24:49.535076 26171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:49.536409 26155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:24:49.536597 26155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "9654461edfff4793bc8e6709609856db"
format_stamp: "Formatted at 2026-08-08 02:24:49 on dist-test-slave-pgkr"
server_key: "303cd59165d250158ada53ad576f6a5e"
server_key_iv: "a168d53e4a50eaa4587fbdacacbd9fa8"
server_key_version: "encryptionkey@0"
I20260808 02:24:49.536823 26155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:49.555502 26155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:49.556381 26155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:49.556679 26155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:49.575858 26155 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:37805
I20260808 02:24:49.575871 26222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:37805 every 8 connection(s)
I20260808 02:24:49.577256 26155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:49.581204 26223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:49.581657 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26155
I20260808 02:24:49.581848 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:49.582535 20380 external_mini_cluster.cc:1428] Setting key 1a16ffbb4ff87a3fa0f079877d454074
I20260808 02:24:49.593772 26223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db: Bootstrap starting.
I20260808 02:24:49.595937 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:49.584489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.19.231.62:37805)
0808 02:24:49.585473 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:49.585484 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:49.585525 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:24:49.587466 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:49.587489 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:49.587959 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:49.588354 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:49.588912 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:49.590168 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:49.591516 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:49.592192 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:49.592416 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:49.593622 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:49.593630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:49.593642 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:24:49.594230 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:49.594305 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:24:49.594330 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:49.594458 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:49.594708 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 02:24:49.595015 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":331,"threads_started":1}
I20260808 02:24:49.597162 26223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:49.598325 26223 log.cc:824] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:49.599917 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:24:49.601374 26223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db: No bootstrap required, opened a new log
I20260808 02:24:49.609408 26223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9654461edfff4793bc8e6709609856db" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37805 } }
I20260808 02:24:49.609809 26223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:49.609910 26223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9654461edfff4793bc8e6709609856db, State: Initialized, Role: FOLLOWER
I20260808 02:24:49.610515 26223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9654461edfff4793bc8e6709609856db" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37805 } }
I20260808 02:24:49.610821 26223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:49.610966 26223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:49.611174 26223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:49.613448 26223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9654461edfff4793bc8e6709609856db" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37805 } }
I20260808 02:24:49.613886 26223 leader_election.cc:302] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9654461edfff4793bc8e6709609856db; no voters:
I20260808 02:24:49.614455 26223 leader_election.cc:288] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:49.614599 26228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:49.615361 26228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [term 1 LEADER]: Becoming Leader. State: Replica: 9654461edfff4793bc8e6709609856db, State: Running, Role: LEADER
I20260808 02:24:49.615991 26228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9654461edfff4793bc8e6709609856db" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37805 } }
I20260808 02:24:49.617098 26223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:49.619081 26230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9654461edfff4793bc8e6709609856db. Latest consensus state: current_term: 1 leader_uuid: "9654461edfff4793bc8e6709609856db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9654461edfff4793bc8e6709609856db" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37805 } } }
I20260808 02:24:49.619191 26229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9654461edfff4793bc8e6709609856db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9654461edfff4793bc8e6709609856db" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37805 } } }
I20260808 02:24:49.619536 26229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:49.619546 26230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:49.622304 26238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:49.627460 26238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:49.636435 26238 catalog_manager.cc:1347] Generated new cluster ID: c3a1838e69f347578215491d81d54d2a
I20260808 02:24:49.636560 26238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:49.648190 26238 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:49.649473 26238 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:49.662379 26238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9654461edfff4793bc8e6709609856db: Generated new TSK 0
I20260808 02:24:49.663293 26238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:24:50.584703 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:49.687158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.19.231.62:37805)
0808 02:24:49.687372 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:49.687380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:49.687819 (+ 439us) server_negotiation.cc:407] Connection header received
0808 02:24:49.688043 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:49.688050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:49.688153 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:49.688336 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0808 02:24:49.688788 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:49.689948 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:49.691519 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:49.692209 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:49.692357 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:49.692850 (+ 493us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 02:24:50.583926 (+891076us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:24:50.584401 (+ 475us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50904: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 02:24:50.584833 26226 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50904: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 02:24:50.584915 26255 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:49.687034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37805 (local address 127.0.0.1:50904)
0808 02:24:49.687712 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:49.687729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:49.687852 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:49.688358 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:49.688362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:49.688372 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 02:24:49.688626 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:49.688635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:49.690143 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:49.690148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:49.691376 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:49.691389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:49.692408 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:49.692412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:49.692533 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:49.692572 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 02:24:50.584293 (+891721us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 02:24:50.584562 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.231.62:37805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":362,"threads_started":1}
I20260808 02:24:50.586175 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26155
I20260808 02:24:50.587354 26218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:50.694224 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26155
2026-08-08T02:24:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1633 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (437 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (429 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:51 dist-test-slave-pgkr krb5kdc[26270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:51 dist-test-slave-pgkr krb5kdc[26270](info): set up 2 sockets
Aug 08 02:24:51 dist-test-slave-pgkr krb5kdc[26270](info): commencing operation
krb5kdc: starting...
W20260808 02:24:52.715714 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:52 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:24:52.982919 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41337
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:41337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:53.340423 26284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:53.340788 26284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:53.340956 26284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:53.348336 26284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:53.348453 26284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:53.348570 26284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:53.348670 26284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:53.348767 26284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:53.359149 26284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:41337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:53.360646 26284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:53.362090 26284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:53.374635 26289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:53.374593 26290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:53.374547 26292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:53.375401 26284 server_base.cc:1034] running on GCE node
I20260808 02:24:53.376209 26284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:53.377698 26284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:53.378932 26284 hybrid_clock.cc:648] HybridClock initialized: now 1786155893378894 us; error 53 us; skew 500 ppm
I20260808 02:24:53.379305 26284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:53 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155893, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:53.382603 26284 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:24:53.384212 26284 webserver.cc:460] Webserver started at http://127.19.231.62:41491/ using document root <none> and password file <none>
I20260808 02:24:53.384936 26284 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:53.385089 26284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:53.385421 26284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:53.388588 26284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "42f87519534943e89d0737f5578b3e54"
format_stamp: "Formatted at 2026-08-08 02:24:53 on dist-test-slave-pgkr"
server_key: "d2970a85a2183b37429c22d8079255b3"
server_key_iv: "d46d35b6632f6b67f4bfbb8f7c275084"
server_key_version: "encryptionkey@0"
I20260808 02:24:53.389431 26284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "42f87519534943e89d0737f5578b3e54"
format_stamp: "Formatted at 2026-08-08 02:24:53 on dist-test-slave-pgkr"
server_key: "d2970a85a2183b37429c22d8079255b3"
server_key_iv: "d46d35b6632f6b67f4bfbb8f7c275084"
server_key_version: "encryptionkey@0"
I20260808 02:24:53.394912 26284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 02:24:53.398408 26299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:53.399791 26284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:53.399986 26284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "42f87519534943e89d0737f5578b3e54"
format_stamp: "Formatted at 2026-08-08 02:24:53 on dist-test-slave-pgkr"
server_key: "d2970a85a2183b37429c22d8079255b3"
server_key_iv: "d46d35b6632f6b67f4bfbb8f7c275084"
server_key_version: "encryptionkey@0"
I20260808 02:24:53.400206 26284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:53.421304 26284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:53.425459 26284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:53.425774 26284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:53.445338 26284 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:41337
I20260808 02:24:53.445349 26351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:41337 every 8 connection(s)
I20260808 02:24:53.446884 26284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:53.450794 26352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:24:53.451447 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26284
I20260808 02:24:53.451706 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:53.452428 20380 external_mini_cluster.cc:1428] Setting key f8bd20af8832111d68b608f22db87f99
I20260808 02:24:53.465583 26352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54: Bootstrap starting.
Aug 08 02:24:53 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:53.468817 26352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:53.470232 26352 log.cc:824] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:53.473816 26352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54: No bootstrap required, opened a new log
I20260808 02:24:53.477561 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:53.454585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.19.231.62:41337)
0808 02:24:53.455497 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:53.455513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:53.455574 (+ 61us) server_negotiation.cc:407] Connection header received
0808 02:24:53.457633 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:53.457674 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:53.458245 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:53.458750 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:53.460086 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:53.462150 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:53.463523 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:53.464591 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:53.464791 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:53.468809 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:53.468822 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:53.468842 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:53.468890 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:53.472851 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:53.473538 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:53.473546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:53.473554 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:53.473772 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:53.474239 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:53.474244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:53.474247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:53.474974 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:53.475295 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:53.475630 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 02:24:53.476124 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":336,"threads_started":1}
I20260808 02:24:53.483047 26352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f87519534943e89d0737f5578b3e54" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41337 } }
I20260808 02:24:53.483384 26352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:53.483459 26352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f87519534943e89d0737f5578b3e54, State: Initialized, Role: FOLLOWER
I20260808 02:24:53.483954 26352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f87519534943e89d0737f5578b3e54" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41337 } }
I20260808 02:24:53.484167 26352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:53.484265 26352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:53.484392 26352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:53.486868 26352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f87519534943e89d0737f5578b3e54" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41337 } }
I20260808 02:24:53.487433 26352 leader_election.cc:302] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f87519534943e89d0737f5578b3e54; no voters:
I20260808 02:24:53.487967 26352 leader_election.cc:288] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:53.488132 26357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:53.488854 26357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [term 1 LEADER]: Becoming Leader. State: Replica: 42f87519534943e89d0737f5578b3e54, State: Running, Role: LEADER
I20260808 02:24:53.489379 26357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f87519534943e89d0737f5578b3e54" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41337 } }
I20260808 02:24:53.490257 26352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:53.492611 26359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f87519534943e89d0737f5578b3e54. Latest consensus state: current_term: 1 leader_uuid: "42f87519534943e89d0737f5578b3e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f87519534943e89d0737f5578b3e54" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41337 } } }
I20260808 02:24:53.493078 26359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:53.493793 26358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f87519534943e89d0737f5578b3e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f87519534943e89d0737f5578b3e54" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41337 } } }
I20260808 02:24:53.494244 26358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:53.496447 26367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:53.501679 26367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:53.510823 26367 catalog_manager.cc:1347] Generated new cluster ID: e341f6c2381b4d0eabcd22d995a32bc5
I20260808 02:24:53.510937 26367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:53.524549 26367 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:53.526463 26367 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:53.537595 26367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f87519534943e89d0737f5578b3e54: Generated new TSK 0
I20260808 02:24:53.538523 26367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:24:53.610708 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41337
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:53.967077 26380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:53.967367 26380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:53.967541 26380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:53.975060 26380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:53.975185 26380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:53.975365 26380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:53.985773 26380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:53.987185 26380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:53.988540 26380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:54.002948 26388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.003007 26385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.003039 26386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:54.004002 26380 server_base.cc:1034] running on GCE node
I20260808 02:24:54.004760 26380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:54.005841 26380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:54.007035 26380 hybrid_clock.cc:648] HybridClock initialized: now 1786155894006964 us; error 58 us; skew 500 ppm
I20260808 02:24:54.007367 26380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:54 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:54.010842 26380 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:24:54.012389 26380 webserver.cc:460] Webserver started at http://127.19.231.1:34445/ using document root <none> and password file <none>
I20260808 02:24:54.013125 26380 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:54.013293 26380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:54.013661 26380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:54.016798 26380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "a46c681f39914177ae50b73669fce907"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-pgkr"
server_key: "27a2d4cb5deb7507a5fc0dab90fefb26"
server_key_iv: "6d54988e1e224658f23660986ecb83f3"
server_key_version: "encryptionkey@0"
I20260808 02:24:54.017611 26380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "a46c681f39914177ae50b73669fce907"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-pgkr"
server_key: "27a2d4cb5deb7507a5fc0dab90fefb26"
server_key_iv: "6d54988e1e224658f23660986ecb83f3"
server_key_version: "encryptionkey@0"
I20260808 02:24:54.022846 26380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 02:24:54.026553 26397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.027979 26380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:54.028183 26380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "a46c681f39914177ae50b73669fce907"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-pgkr"
server_key: "27a2d4cb5deb7507a5fc0dab90fefb26"
server_key_iv: "6d54988e1e224658f23660986ecb83f3"
server_key_version: "encryptionkey@0"
I20260808 02:24:54.028404 26380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:54.069123 26380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:54.073447 26380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:54.073787 26380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:54.074898 26380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:54.076388 26380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:54.076499 26380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.076682 26380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:54.076764 26380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.111303 26380 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:34671
I20260808 02:24:54.111367 26514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:34671 every 8 connection(s)
I20260808 02:24:54.112658 26380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:54.117973 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26380
I20260808 02:24:54.118196 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:54.119077 20380 external_mini_cluster.cc:1428] Setting key 0d88fee177c15f2d8fd62781bad4d10c
Aug 08 02:24:54 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:54.136404 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.115272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:60701 (local address 127.19.231.62:41337)
0808 02:24:54.115720 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:54.115724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:54.116798 (+ 1074us) server_negotiation.cc:407] Connection header received
0808 02:24:54.119320 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:54.119324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:54.119407 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:54.119501 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:54.121042 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.121758 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.124303 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.124723 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.124816 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.129038 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:54.129045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:54.129049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:54.129086 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:54.132521 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.133501 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.133522 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.133525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.133673 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.134210 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.134223 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.134226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.134550 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:54.134801 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:54.135690 (+ 889us) server_negotiation.cc:299] Negotiation successful
0808 02:24:54.135962 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":214,"threads_started":1}
I20260808 02:24:54.137539 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.115931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:41337 (local address 127.19.231.1:60701)
0808 02:24:54.116627 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:54.116672 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:54.119009 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:54.119814 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:54.119822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:54.120324 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:54.120882 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.120898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.121925 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.121933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.124143 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.124160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.124945 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.124954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.125272 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.126060 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:54.126097 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:54.128814 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:54.132679 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.132689 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.132706 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.133349 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.133813 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.133820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.133824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.134062 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.134788 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:54.134797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:54.135191 (+ 394us) client_negotiation.cc:769] Sending connection context
0808 02:24:54.135524 (+ 333us) client_negotiation.cc:240] Negotiation successful
0808 02:24:54.136131 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":292,"threads_started":1}
I20260808 02:24:54.139371 26515 heartbeater.cc:347] Connected to a master server at 127.19.231.62:41337
I20260808 02:24:54.139722 26515 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:54.140676 26515 heartbeater.cc:511] Master 127.19.231.62:41337 requested a full tablet report, sending...
I20260808 02:24:54.143380 26317 ts_manager.cc:194] Registered new tserver with Master: a46c681f39914177ae50b73669fce907 (127.19.231.1:34671)
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
I20260808 02:24:54.145574 26317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:60701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:24:54.178293 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41337
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:54.514904 26521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:54.515188 26521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:54.515302 26521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:54.522423 26521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:54.522533 26521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:54.522708 26521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:54.532781 26521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:54.534080 26521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:54.535558 26521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:54.548851 26529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.548830 26526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:54.548799 26527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:54.549685 26521 server_base.cc:1034] running on GCE node
I20260808 02:24:54.550352 26521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:54.551394 26521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:54.552625 26521 hybrid_clock.cc:648] HybridClock initialized: now 1786155894552576 us; error 45 us; skew 500 ppm
I20260808 02:24:54.552975 26521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:54 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:54.556257 26521 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:24:54.557799 26521 webserver.cc:460] Webserver started at http://127.19.231.2:37553/ using document root <none> and password file <none>
I20260808 02:24:54.558488 26521 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:54.558689 26521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:54.559011 26521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:54.561939 26521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "af596665b462491e87173ba77d590d6b"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-pgkr"
server_key: "3bf238e889d78d6eea59df0d25ea2fb1"
server_key_iv: "4ce7744c3f35c9a2bb90e7d1b265b319"
server_key_version: "encryptionkey@0"
I20260808 02:24:54.562798 26521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "af596665b462491e87173ba77d590d6b"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-pgkr"
server_key: "3bf238e889d78d6eea59df0d25ea2fb1"
server_key_iv: "4ce7744c3f35c9a2bb90e7d1b265b319"
server_key_version: "encryptionkey@0"
I20260808 02:24:54.567838 26521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:24:54.571385 26536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.572764 26521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:54.572896 26521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "af596665b462491e87173ba77d590d6b"
format_stamp: "Formatted at 2026-08-08 02:24:54 on dist-test-slave-pgkr"
server_key: "3bf238e889d78d6eea59df0d25ea2fb1"
server_key_iv: "4ce7744c3f35c9a2bb90e7d1b265b319"
server_key_version: "encryptionkey@0"
I20260808 02:24:54.573056 26521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:54.593156 26521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:54.597641 26521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:54.597980 26521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:54.599010 26521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:54.600327 26521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:54.600400 26521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.600576 26521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:54.600629 26521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:54.635938 26521 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:34491
I20260808 02:24:54.636080 26656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:34491 every 8 connection(s)
I20260808 02:24:54.637351 26521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:54.642377 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26521
I20260808 02:24:54.642566 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:54.643054 20380 external_mini_cluster.cc:1428] Setting key 11d812c2a3fda744c073f5270fc0059b
Aug 08 02:24:54 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:54.660020 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.640108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:56451 (local address 127.19.231.62:41337)
0808 02:24:54.640634 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:54.640638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:54.641464 (+ 826us) server_negotiation.cc:407] Connection header received
0808 02:24:54.643729 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:54.643733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:54.643803 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:54.643929 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:54.645945 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.646797 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.649270 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.649753 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.649866 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.654188 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:54.654197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:54.654205 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:54.654252 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:54.656951 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.657683 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.657688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.657692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.657829 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:54.658271 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:54.658278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:54.658281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:54.658544 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:54.658737 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:54.659398 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 02:24:54.659625 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":255,"threads_started":1}
I20260808 02:24:54.661010 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:54.640520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:41337 (local address 127.19.231.2:56451)
0808 02:24:54.641277 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:54.641330 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:54.643395 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:54.644223 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:54.644235 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:54.644975 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:54.645749 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.645775 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.646961 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.646970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.649107 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:54.649124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:54.649997 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:54.650007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:54.650345 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:54.651112 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:54.651151 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:54.653922 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:54.657097 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.657106 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.657122 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.657549 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.657946 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:54.657951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:54.657954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:54.658148 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:54.658696 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:54.658704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:54.659041 (+ 337us) client_negotiation.cc:769] Sending connection context
0808 02:24:54.659321 (+ 280us) client_negotiation.cc:240] Negotiation successful
0808 02:24:54.659846 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":296,"threads_started":1}
I20260808 02:24:54.662484 26657 heartbeater.cc:347] Connected to a master server at 127.19.231.62:41337
I20260808 02:24:54.662842 26657 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:54.663578 26657 heartbeater.cc:511] Master 127.19.231.62:41337 requested a full tablet report, sending...
I20260808 02:24:54.665120 26317 ts_manager.cc:194] Registered new tserver with Master: af596665b462491e87173ba77d590d6b (127.19.231.2:34491)
I20260808 02:24:54.666482 26317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:56451
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:24:54.699870 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41337
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:55.036024 26663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:55.036293 26663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:55.036386 26663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:55.043351 26663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:55.043452 26663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:55.043561 26663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:55.053161 26663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:41337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:55.054401 26663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:55.056046 26663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:55.070140 26669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:55.070163 26668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:55.071003 26663 server_base.cc:1034] running on GCE node
W20260808 02:24:55.071017 26672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:55.071777 26663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:55.072738 26663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:55.073974 26663 hybrid_clock.cc:648] HybridClock initialized: now 1786155895073929 us; error 37 us; skew 500 ppm
I20260808 02:24:55.074293 26663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.077529 26663 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:24:55.079310 26663 webserver.cc:460] Webserver started at http://127.19.231.3:40875/ using document root <none> and password file <none>
I20260808 02:24:55.080018 26663 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:55.080168 26663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:55.080490 26663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:55.083642 26663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "b8e30b7230f746f79a133ff982c8f860"
format_stamp: "Formatted at 2026-08-08 02:24:55 on dist-test-slave-pgkr"
server_key: "0ed38c9c23d57e4aade0c7ee4b92054f"
server_key_iv: "95c46b53c66e6eb067f7a780850910ab"
server_key_version: "encryptionkey@0"
I20260808 02:24:55.084439 26663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "b8e30b7230f746f79a133ff982c8f860"
format_stamp: "Formatted at 2026-08-08 02:24:55 on dist-test-slave-pgkr"
server_key: "0ed38c9c23d57e4aade0c7ee4b92054f"
server_key_iv: "95c46b53c66e6eb067f7a780850910ab"
server_key_version: "encryptionkey@0"
I20260808 02:24:55.089787 26663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:24:55.093657 26679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.095160 26663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:55.095291 26663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "b8e30b7230f746f79a133ff982c8f860"
format_stamp: "Formatted at 2026-08-08 02:24:55 on dist-test-slave-pgkr"
server_key: "0ed38c9c23d57e4aade0c7ee4b92054f"
server_key_iv: "95c46b53c66e6eb067f7a780850910ab"
server_key_version: "encryptionkey@0"
I20260808 02:24:55.095458 26663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:55.130882 26663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:55.135170 26663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:55.135419 26663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:55.136358 26663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:55.137681 26663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:55.137760 26663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.137851 26663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:55.137895 26663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:55.148838 26515 heartbeater.cc:503] Master 127.19.231.62:41337 was elected leader, sending a full tablet report...
I20260808 02:24:55.172518 26663 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:40285
I20260808 02:24:55.172607 26798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:40285 every 8 connection(s)
I20260808 02:24:55.173846 26663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:55.174249 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26663
I20260808 02:24:55.174435 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:55.175128 20380 external_mini_cluster.cc:1428] Setting key 24f9a6b609ff546087caedc461b82f65
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:24:55.195230 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.176661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:45333 (local address 127.19.231.62:41337)
0808 02:24:55.177213 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.177221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.178799 (+ 1578us) server_negotiation.cc:407] Connection header received
0808 02:24:55.180330 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.180336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.180420 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.180546 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.181990 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.182870 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.184369 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.184774 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.184909 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.188225 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.188230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.188233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.188266 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.192308 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.192995 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.192999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.193002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.193165 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.193530 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.193533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.193535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.193918 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.194052 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.194782 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.194949 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":401,"threads_started":1}
I20260808 02:24:55.196190 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.177401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:41337 (local address 127.19.231.3:45333)
0808 02:24:55.178279 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:55.178373 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:55.180074 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:55.180784 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:55.180792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:55.181283 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:55.181827 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.181844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.183026 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.183031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.184223 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:55.184235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.184999 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.185007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:55.185229 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.185850 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:55.185877 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:55.188035 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:55.192426 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.192432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.192445 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.192850 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.193261 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:55.193265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:55.193267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:55.193440 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:55.194066 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:55.194076 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:55.194403 (+ 327us) client_negotiation.cc:769] Sending connection context
0808 02:24:55.194682 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 02:24:55.195052 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":393,"threads_started":1}
I20260808 02:24:55.197656 26799 heartbeater.cc:347] Connected to a master server at 127.19.231.62:41337
I20260808 02:24:55.197954 26799 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:55.198720 26799 heartbeater.cc:511] Master 127.19.231.62:41337 requested a full tablet report, sending...
I20260808 02:24:55.200317 26317 ts_manager.cc:194] Registered new tserver with Master: b8e30b7230f746f79a133ff982c8f860 (127.19.231.3:40285)
I20260808 02:24:55.201129 26317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:45333
I20260808 02:24:55.204142 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/test-admin.keytab.
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.236963 20380 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 02:24:55.242763 26807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:55.240133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.19.231.1:34671)
0808 02:24:55.240809 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.240816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.240844 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:24:55.240916 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.240922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.241129 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.241269 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.242253 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47012: BlockingRecv error: recv got EOF from 127.0.0.1:47012 (error 108)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":365,"threads_started":1}
W20260808 02:24:55.242910 20380 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.245036 20380 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.246825 20380 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:55.264581 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.249104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.19.231.1:34671)
0808 02:24:55.250748 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.250754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.250774 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:24:55.250840 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.250845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.250921 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.251036 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.252224 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.253213 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.254476 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.255128 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.255231 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.258909 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.258920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.258934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.258974 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.261759 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.262442 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.262449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.262451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.262574 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.263008 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.263013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.263016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.263407 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.263710 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.264053 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.264307 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485}
W20260808 02:24:55.270341 26807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:55.268473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.19.231.1:34671)
0808 02:24:55.268690 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.268697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.269000 (+ 303us) server_negotiation.cc:407] Connection header received
0808 02:24:55.269232 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.269237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.269305 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.269401 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.270096 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47034: BlockingRecv error: recv got EOF from 127.0.0.1:47034 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 02:24:55.270816 20380 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc)
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.272315 20380 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.273693 20380 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:55.288389 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.274970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.19.231.1:34671)
0808 02:24:55.275167 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.275175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.275467 (+ 292us) server_negotiation.cc:407] Connection header received
0808 02:24:55.275644 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.275657 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.275722 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.275820 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.276821 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.277506 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.278759 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.279482 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.279698 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.283425 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.283430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.283433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.283464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.285866 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.286493 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.286500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.286503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.286697 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.287075 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.287080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.287084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.287360 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.287635 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.287900 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.288105 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 02:24:55.292948 26807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:24:55.291110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.19.231.1:34671)
0808 02:24:55.291319 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.291326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.291643 (+ 317us) server_negotiation.cc:407] Connection header received
0808 02:24:55.291832 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.291838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.291915 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.292015 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.292728 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52802: BlockingRecv error: recv got EOF from 127.0.0.1:52802 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 02:24:55.293165 20380 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786155839876607-20380-0/krb5kdc/krb5cc' of type FILE
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.294926 20380 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:55.296263 20380 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:24:55.310499 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.297571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.19.231.1:34671)
0808 02:24:55.297792 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.297797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.297989 (+ 192us) server_negotiation.cc:407] Connection header received
0808 02:24:55.298108 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.298111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.298166 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.298244 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.299388 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.300146 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.301342 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.301750 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.301850 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.305681 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.305686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.305689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.305719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.308187 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.308774 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.308777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.308779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.308898 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.309291 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.309294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.309296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.309545 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.309792 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.309979 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.310170 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 02:24:55.312191 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26380
I20260808 02:24:55.323908 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.312885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.19.231.1:34671)
0808 02:24:55.313071 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.313077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.313364 (+ 287us) server_negotiation.cc:407] Connection header received
0808 02:24:55.313535 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.313541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.313604 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.313704 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.314651 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.315311 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.316541 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.316939 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.317080 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.318985 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.318990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.318993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.319020 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.321373 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.321989 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.321996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.321999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.322132 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.322501 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.322506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.322510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.322823 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.323089 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.323392 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.323590 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:24:55.324446 26510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:55.413173 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26380
I20260808 02:24:55.436918 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26521
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:24:55.454450 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.437974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.19.231.2:34491)
0808 02:24:55.438672 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.438685 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.438713 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:24:55.438776 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.438782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.438999 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.439145 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.440126 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.441223 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.442402 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.443150 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.443271 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.447246 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.447261 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.447279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.447336 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.451347 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.452070 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.452081 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.452085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.452271 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.452650 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.452656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.452660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.453137 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.453456 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.453693 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.453980 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":332,"threads_started":1}
I20260808 02:24:55.456028 26651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:55.541211 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26521
I20260808 02:24:55.561882 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26663
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:24:55.578563 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:55.563032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.19.231.3:40285)
0808 02:24:55.563739 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:55.563745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:55.563775 (+ 30us) server_negotiation.cc:407] Connection header received
0808 02:24:55.563841 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:55.563847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:55.564056 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:55.564190 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:55.565348 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.566369 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.567813 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:55.568381 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:55.568480 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:55.572729 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:55.572746 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:55.572765 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:55.572817 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:55.575834 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.576612 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.576618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.576621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.576757 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:55.577137 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:55.577141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:55.577143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:55.577463 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:55.577710 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:55.577972 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 02:24:55.578264 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":336,"threads_started":1}
I20260808 02:24:55.579922 26794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:55.666767 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26663
I20260808 02:24:55.686789 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26284
I20260808 02:24:55.688028 26347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:24:55.797642 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26284
2026-08-08T02:24:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4239 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26828](info): set up 2 sockets
Aug 08 02:24:55 dist-test-slave-pgkr krb5kdc[26828](info): commencing operation
krb5kdc: starting...
W20260808 02:24:56.959192 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:24:57 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:24:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab.
Entry for principal oryx/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab.
I20260808 02:24:57.226552 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:46797
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab
--principal=oryx/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:57.580096 26841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:57.580451 26841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:57.580617 26841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:57.587940 26841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:57.588088 26841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:57.588209 26841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:57.588310 26841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:57.588398 26841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:57.598299 26841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab
--principal=oryx/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:57.599766 26841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:57.601192 26841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:57.613480 26846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:57.613454 26847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:57.613857 26849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:57.614369 26841 server_base.cc:1034] running on GCE node
I20260808 02:24:57.615258 26841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:57.616711 26841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:57.617952 26841 hybrid_clock.cc:648] HybridClock initialized: now 1786155897617904 us; error 60 us; skew 500 ppm
I20260808 02:24:57.618311 26841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:57 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:57.626468 26841 init.cc:377] Logged in from keytab as oryx/127.19.231.62@KRBTEST.COM (short username oryx)
I20260808 02:24:57.628090 26841 webserver.cc:460] Webserver started at http://127.19.231.62:40007/ using document root <none> and password file <none>
I20260808 02:24:57.628820 26841 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:57.628976 26841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:57.629300 26841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:57.632464 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "63275c4186764355a53e17d49746ffe9"
format_stamp: "Formatted at 2026-08-08 02:24:57 on dist-test-slave-pgkr"
server_key: "164503666129457cc7bc91e999b5dd77"
server_key_iv: "8ba792af795024ee3b6808b6fc5b67dd"
server_key_version: "encryptionkey@0"
I20260808 02:24:57.633250 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "63275c4186764355a53e17d49746ffe9"
format_stamp: "Formatted at 2026-08-08 02:24:57 on dist-test-slave-pgkr"
server_key: "164503666129457cc7bc91e999b5dd77"
server_key_iv: "8ba792af795024ee3b6808b6fc5b67dd"
server_key_version: "encryptionkey@0"
I20260808 02:24:57.638815 26841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:24:57.642218 26856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:57.643606 26841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:57.643807 26841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "63275c4186764355a53e17d49746ffe9"
format_stamp: "Formatted at 2026-08-08 02:24:57 on dist-test-slave-pgkr"
server_key: "164503666129457cc7bc91e999b5dd77"
server_key_iv: "8ba792af795024ee3b6808b6fc5b67dd"
server_key_version: "encryptionkey@0"
I20260808 02:24:57.644012 26841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:57.666448 26841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:57.670677 26841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:57.670980 26841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:57.690440 26841 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:46797
I20260808 02:24:57.690474 26911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:46797 every 8 connection(s)
I20260808 02:24:57.691917 26841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:24:57.693416 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26841
I20260808 02:24:57.693634 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:24:57.694473 20380 external_mini_cluster.cc:1428] Setting key 3c6f294c4b036f56ed96bbc3b39ff75d
I20260808 02:24:57.697028 26912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:24:57 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:24:57.710832 26912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: Bootstrap starting.
I20260808 02:24:57.715307 26912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: Neither blocks nor log segments found. Creating new log.
I20260808 02:24:57.716675 26912 log.cc:824] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:24:57.719841 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:57.696677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.19.231.62:46797)
0808 02:24:57.697525 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:57.697537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:57.697577 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:24:57.699306 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:57.699327 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:57.699719 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:57.700057 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:57.701995 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.703446 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.705251 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:57.705985 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:57.706159 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:57.711012 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:57.711023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:57.711041 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:57.711084 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:57.715080 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:57.715835 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:57.715844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:57.715852 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:57.716057 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:57.716447 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:57.716455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:57.716460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:57.717238 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:57.717557 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:57.717991 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 02:24:57.718425 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":410,"threads_started":1}
I20260808 02:24:57.720453 26912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: No bootstrap required, opened a new log
I20260808 02:24:57.728446 26912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } }
I20260808 02:24:57.728787 26912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:24:57.728869 26912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63275c4186764355a53e17d49746ffe9, State: Initialized, Role: FOLLOWER
I20260808 02:24:57.729441 26912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } }
I20260808 02:24:57.729653 26912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:24:57.729756 26912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:24:57.729883 26912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:24:57.732369 26912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } }
I20260808 02:24:57.732818 26912 leader_election.cc:302] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63275c4186764355a53e17d49746ffe9; no voters:
I20260808 02:24:57.733390 26912 leader_election.cc:288] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:24:57.733546 26917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:24:57.734225 26917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 1 LEADER]: Becoming Leader. State: Replica: 63275c4186764355a53e17d49746ffe9, State: Running, Role: LEADER
I20260808 02:24:57.734833 26917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } }
I20260808 02:24:57.735616 26912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:24:57.737897 26919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63275c4186764355a53e17d49746ffe9. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } }
I20260808 02:24:57.738338 26919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:57.739219 26918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } }
I20260808 02:24:57.739624 26918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:24:57.740157 26923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:24:57.746650 26923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:24:57.756026 26923 catalog_manager.cc:1347] Generated new cluster ID: d25a7a1b302a4210b8d52e91de56df63
I20260808 02:24:57.756165 26923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:24:57.768599 26923 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:24:57.769917 26923 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:24:57.780689 26923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: Generated new TSK 0
I20260808 02:24:57.781540 26923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab.
Entry for principal oryx/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab.
I20260808 02:24:57.847385 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:46797
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab
--principal=oryx/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:58.200006 26938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:58.200363 26938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:58.200562 26938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:58.208007 26938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:58.208173 26938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:58.208354 26938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:24:58.218722 26938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab
--principal=oryx/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:58.220081 26938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:58.221604 26938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:58.235584 26944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:58.235625 26946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:58.235671 26943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:58.236447 26938 server_base.cc:1034] running on GCE node
I20260808 02:24:58.237196 26938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:58.238245 26938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:58.239459 26938 hybrid_clock.cc:648] HybridClock initialized: now 1786155898239417 us; error 32 us; skew 500 ppm
I20260808 02:24:58.239771 26938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:58 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155898, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:58.243588 26938 init.cc:377] Logged in from keytab as oryx/127.19.231.1@KRBTEST.COM (short username oryx)
I20260808 02:24:58.245136 26938 webserver.cc:460] Webserver started at http://127.19.231.1:36317/ using document root <none> and password file <none>
I20260808 02:24:58.245800 26938 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:58.245916 26938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:58.246178 26938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:58.249483 26938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "92f14ffc1fc14e6ea70b96582bc99e71"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "1bcc9c59dc91b67e6e489d32c6188f90"
server_key_iv: "01f16f0899e93b38bac810e2ec05eb50"
server_key_version: "encryptionkey@0"
I20260808 02:24:58.250298 26938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "92f14ffc1fc14e6ea70b96582bc99e71"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "1bcc9c59dc91b67e6e489d32c6188f90"
server_key_iv: "01f16f0899e93b38bac810e2ec05eb50"
server_key_version: "encryptionkey@0"
I20260808 02:24:58.255635 26938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:24:58.259222 26953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:58.260671 26938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 02:24:58.260840 26938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "92f14ffc1fc14e6ea70b96582bc99e71"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "1bcc9c59dc91b67e6e489d32c6188f90"
server_key_iv: "01f16f0899e93b38bac810e2ec05eb50"
server_key_version: "encryptionkey@0"
I20260808 02:24:58.261061 26938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:58.291430 26938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:58.295948 26938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:58.296268 26938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:58.297331 26938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:58.298833 26938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:58.298909 26938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:58.299048 26938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:58.299113 26938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:58.334877 26938 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:42421
I20260808 02:24:58.334996 27072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:42421 every 8 connection(s)
I20260808 02:24:58.336355 26938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:24:58.346235 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 26938
I20260808 02:24:58.346477 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:24:58.347412 20380 external_mini_cluster.cc:1428] Setting key 31e6b673f6bb9c544462b718ec32a5ba
Aug 08 02:24:58 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155898, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:24:58.359124 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:58.339210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:45203 (local address 127.19.231.62:46797)
0808 02:24:58.339719 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:58.339726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:58.340891 (+ 1165us) server_negotiation.cc:407] Connection header received
0808 02:24:58.342835 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:58.342841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:58.342912 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:58.343015 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:58.344496 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.345303 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.346991 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.347754 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.347902 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:58.352603 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:58.352608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:58.352612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:58.352644 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:58.355506 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:58.356552 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:58.356556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:58.356558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:58.356684 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:58.357228 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:58.357231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:58.357233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:58.357499 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:58.357616 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:58.358685 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0808 02:24:58.358857 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260808 02:24:58.360872 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:58.339967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.1:45203)
0808 02:24:58.340720 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:58.340772 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:58.342509 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:58.343217 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:58.343225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:58.343770 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:58.344333 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:58.344351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.345466 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.345470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:58.346830 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:58.346843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.348017 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.348023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:58.348321 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:58.349068 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:58.349106 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:58.352376 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:58.355659 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:58.355671 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:58.355692 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:58.356408 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:58.356806 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:58.356814 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:58.356821 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:58.357093 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:58.357641 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:58.357653 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:58.358199 (+ 546us) client_negotiation.cc:769] Sending connection context
0808 02:24:58.358540 (+ 341us) client_negotiation.cc:240] Negotiation successful
0808 02:24:58.359224 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260808 02:24:58.362794 27073 heartbeater.cc:347] Connected to a master server at 127.19.231.62:46797
I20260808 02:24:58.363127 27073 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:58.363937 27073 heartbeater.cc:511] Master 127.19.231.62:46797 requested a full tablet report, sending...
I20260808 02:24:58.366644 26874 ts_manager.cc:194] Registered new tserver with Master: 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421)
I20260808 02:24:58.368919 26874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:45203
WARNING: no policy specified for oryx/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab.
Entry for principal oryx/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab.
I20260808 02:24:58.410550 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:46797
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab
--principal=oryx/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:58.746064 27079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:58.746344 27079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:58.746438 27079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:58.753901 27079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:58.754020 27079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:58.754136 27079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:24:58.764243 27079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab
--principal=oryx/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:58.765727 27079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:58.767091 27079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:58.780335 27087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:58.780404 27085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:58.780432 27084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:58.781605 27079 server_base.cc:1034] running on GCE node
I20260808 02:24:58.782251 27079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:58.783300 27079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:58.784497 27079 hybrid_clock.cc:648] HybridClock initialized: now 1786155898784446 us; error 48 us; skew 500 ppm
I20260808 02:24:58.784823 27079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:58 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155898, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:58.788074 27079 init.cc:377] Logged in from keytab as oryx/127.19.231.2@KRBTEST.COM (short username oryx)
I20260808 02:24:58.789556 27079 webserver.cc:460] Webserver started at http://127.19.231.2:45369/ using document root <none> and password file <none>
I20260808 02:24:58.790261 27079 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:58.790416 27079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:58.790819 27079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:58.793735 27079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "56d0db5c0d25f6174c4a405cc1cb4be8"
server_key_iv: "029a6d786c7bb45654e347c90dfe8a77"
server_key_version: "encryptionkey@0"
I20260808 02:24:58.794528 27079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "56d0db5c0d25f6174c4a405cc1cb4be8"
server_key_iv: "029a6d786c7bb45654e347c90dfe8a77"
server_key_version: "encryptionkey@0"
I20260808 02:24:58.799687 27079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:24:58.803913 27094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:58.805990 27079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20260808 02:24:58.806233 27079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "56d0db5c0d25f6174c4a405cc1cb4be8"
server_key_iv: "029a6d786c7bb45654e347c90dfe8a77"
server_key_version: "encryptionkey@0"
I20260808 02:24:58.806550 27079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:58.824384 27079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:58.828384 27079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:58.828698 27079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:58.829704 27079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:58.831207 27079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:58.831283 27079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:58.831468 27079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:58.831521 27079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:58.867120 27079 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:36681
I20260808 02:24:58.867213 27209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:36681 every 8 connection(s)
I20260808 02:24:58.868443 27079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:24:58.876832 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 27079
I20260808 02:24:58.877045 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:24:58.877580 20380 external_mini_cluster.cc:1428] Setting key 7cfaf176270fdc3d66606a76ebe161c2
Aug 08 02:24:58 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155898, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:24:58.889525 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:58.871347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:58179 (local address 127.19.231.62:46797)
0808 02:24:58.871808 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:58.871813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:58.872804 (+ 991us) server_negotiation.cc:407] Connection header received
0808 02:24:58.874595 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:58.874601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:58.874727 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:58.874853 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:58.876329 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.877223 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.879010 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.879580 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.879722 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:58.883874 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:58.883879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:58.883882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:58.883913 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:58.886476 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:58.887234 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:58.887242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:58.887246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:58.887379 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:58.887816 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:58.887820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:58.887825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:58.888079 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:58.888243 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:58.888899 (+ 656us) server_negotiation.cc:299] Negotiation successful
0808 02:24:58.889132 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":218,"threads_started":1}
I20260808 02:24:58.890501 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:58.871932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.2:58179)
0808 02:24:58.872638 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:58.872688 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:58.874335 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:58.875091 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:58.875099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:58.875610 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:58.876164 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:58.876181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.877381 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.877385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:58.878871 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:58.878883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:58.879825 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:58.879833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:58.880154 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:58.880878 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:58.880911 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:58.883652 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:58.886656 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:58.886666 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:58.886681 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:58.887101 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:58.887496 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:58.887501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:58.887505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:58.887695 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:58.888218 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:58.888226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:58.888555 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 02:24:58.888806 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 02:24:58.889349 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":284,"threads_started":1}
I20260808 02:24:58.891965 27210 heartbeater.cc:347] Connected to a master server at 127.19.231.62:46797
I20260808 02:24:58.892310 27210 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:58.893261 27210 heartbeater.cc:511] Master 127.19.231.62:46797 requested a full tablet report, sending...
I20260808 02:24:58.894860 26874 ts_manager.cc:194] Registered new tserver with Master: fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681)
I20260808 02:24:58.895681 26874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:58179
WARNING: no policy specified for oryx/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab.
Entry for principal oryx/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab.
I20260808 02:24:58.937703 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:46797
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab
--principal=oryx/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:24:59.282346 27217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:59.282640 27217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:59.282806 27217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:59.289929 27217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:59.290035 27217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:59.290194 27217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:24:59.299865 27217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab
--principal=oryx/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:24:59.301226 27217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:59.302958 27217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:59.316141 27222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.316226 27223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.316309 27225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:59.320051 27217 server_base.cc:1034] running on GCE node
I20260808 02:24:59.320744 27217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:59.321744 27217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:59.322986 27217 hybrid_clock.cc:648] HybridClock initialized: now 1786155899322937 us; error 60 us; skew 500 ppm
I20260808 02:24:59.323325 27217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:59 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:59.326586 27217 init.cc:377] Logged in from keytab as oryx/127.19.231.3@KRBTEST.COM (short username oryx)
I20260808 02:24:59.328240 27217 webserver.cc:460] Webserver started at http://127.19.231.3:33827/ using document root <none> and password file <none>
I20260808 02:24:59.328987 27217 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:59.329145 27217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:59.329457 27217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:59.332576 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "a5b93a6b19e943eea6354fc7971e9e78"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "e85d08ec8f264251f6a4ee1f5d5c93c8"
server_key_iv: "7397d90b5e7efc0a3195c79b66b637eb"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.333421 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b93a6b19e943eea6354fc7971e9e78"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "e85d08ec8f264251f6a4ee1f5d5c93c8"
server_key_iv: "7397d90b5e7efc0a3195c79b66b637eb"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.339320 27217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 02:24:59.342839 27232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.344234 27217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:24:59.344412 27217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "a5b93a6b19e943eea6354fc7971e9e78"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "e85d08ec8f264251f6a4ee1f5d5c93c8"
server_key_iv: "7397d90b5e7efc0a3195c79b66b637eb"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.344668 27217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:59.370782 27217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:59.372401 27073 heartbeater.cc:503] Master 127.19.231.62:46797 was elected leader, sending a full tablet report...
I20260808 02:24:59.375538 27217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:59.375783 27217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:24:59.376669 27217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:24:59.378018 27217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:24:59.378100 27217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.378185 27217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:24:59.378228 27217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.409920 27217 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:36813
I20260808 02:24:59.409962 27348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:36813 every 8 connection(s)
I20260808 02:24:59.411347 27217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:24:59.414299 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 27217
I20260808 02:24:59.414486 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:24:59.415019 20380 external_mini_cluster.cc:1428] Setting key c27722c6a50c687bdc8ec4357776b9e2
Aug 08 02:24:59 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:24:59.431926 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.414332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:60563 (local address 127.19.231.62:46797)
0808 02:24:59.414878 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0808 02:24:59.414886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:24:59.416011 (+ 1125us) server_negotiation.cc:407] Connection header received
0808 02:24:59.417843 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:24:59.417850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:24:59.417936 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:24:59.418033 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:24:59.419548 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.420302 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.421797 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.422187 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.422280 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.425625 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:24:59.425630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:24:59.425633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:24:59.425663 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:24:59.428713 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.429590 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.429594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.429596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.429732 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:24:59.430251 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:24:59.430255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:24:59.430257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:24:59.430605 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:24:59.430798 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 02:24:59.431455 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 02:24:59.431656 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":271,"threads_started":1}
I20260808 02:24:59.432998 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:24:59.414980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.3:60563)
0808 02:24:59.415861 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0808 02:24:59.415922 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:24:59.417599 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:24:59.418282 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:24:59.418291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:24:59.418866 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:24:59.419407 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.419424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.420435 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.420439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.421646 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:24:59.421677 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:24:59.422373 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:24:59.422377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:24:59.422564 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:24:59.423327 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:24:59.423348 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:24:59.425436 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:24:59.428844 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.428853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.428870 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.429450 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.429851 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:24:59.429856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:24:59.429860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:24:59.430110 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:24:59.430799 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:24:59.430805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:24:59.431111 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 02:24:59.431348 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 02:24:59.431860 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":398,"threads_started":1}
I20260808 02:24:59.434351 27349 heartbeater.cc:347] Connected to a master server at 127.19.231.62:46797
I20260808 02:24:59.434676 27349 heartbeater.cc:464] Registering TS with master...
I20260808 02:24:59.435406 27349 heartbeater.cc:511] Master 127.19.231.62:46797 requested a full tablet report, sending...
I20260808 02:24:59.436908 26874 ts_manager.cc:194] Registered new tserver with Master: a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813)
I20260808 02:24:59.437716 26874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:60563
I20260808 02:24:59.442700 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.231.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.61.keytab.
Entry for principal oryx/127.19.231.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.61.keytab.
I20260808 02:24:59.501929 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:46471
--webserver_interface=127.19.231.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.61.keytab
--principal=oryx/127.19.231.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:46797,127.19.231.61:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:24:59.899075 27210 heartbeater.cc:503] Master 127.19.231.62:46797 was elected leader, sending a full tablet report...
W20260808 02:24:59.917716 27355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:24:59.918042 27355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:24:59.918155 27355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:24:59.925781 27355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:24:59.925905 27355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:24:59.925969 27355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:24:59.926036 27355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:24:59.926088 27355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:24:59.936800 27355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:46797,127.19.231.61:46471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.61.keytab
--principal=oryx/127.19.231.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.231.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
W20260808 02:24:59.937229 27355 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.231.62:46797,127.19.231.61:46471'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 02:24:59.938352 27355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:24:59.939807 27355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:24:59.952522 27360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.952591 27364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:24:59.952622 27361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:24:59.953106 27355 server_base.cc:1034] running on GCE node
I20260808 02:24:59.953963 27355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:24:59.955547 27355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:24:59.956828 27355 hybrid_clock.cc:648] HybridClock initialized: now 1786155899956759 us; error 39 us; skew 500 ppm
I20260808 02:24:59.957211 27355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:24:59 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:24:59.960563 27355 init.cc:377] Logged in from keytab as oryx/127.19.231.61@KRBTEST.COM (short username oryx)
I20260808 02:24:59.962225 27355 webserver.cc:460] Webserver started at http://127.19.231.61:42241/ using document root <none> and password file <none>
I20260808 02:24:59.963063 27355 fs_manager.cc:359] Metadata directory not provided
I20260808 02:24:59.963207 27355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:24:59.963580 27355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:24:59.967136 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/instance:
uuid: "7343c0d9e76b42198b991afcbeca6199"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "5d21edc4cc0c553b65043259d04af6c9"
server_key_iv: "cf1cba1e2f3785496a9baa4f78457f12"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.968017 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal/instance:
uuid: "7343c0d9e76b42198b991afcbeca6199"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "5d21edc4cc0c553b65043259d04af6c9"
server_key_iv: "cf1cba1e2f3785496a9baa4f78457f12"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.973513 27355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:24:59.977272 27372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:24:59.978794 27355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:24:59.979000 27355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
uuid: "7343c0d9e76b42198b991afcbeca6199"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "5d21edc4cc0c553b65043259d04af6c9"
server_key_iv: "cf1cba1e2f3785496a9baa4f78457f12"
server_key_version: "encryptionkey@0"
I20260808 02:24:59.979264 27355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:24:59.994074 27355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:24:59.998450 27355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:24:59.998795 27355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:00.019277 27355 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.61:46471
I20260808 02:25:00.019285 27424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.61:46471 every 8 connection(s)
I20260808 02:25:00.020821 27355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
I20260808 02:25:00.024752 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:00.028734 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 27355
I20260808 02:25:00.032682 27425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:25:00 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.61@KRBTEST.COM
Aug 08 02:25:00 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.61@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:00.054754 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.037054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.19.231.62:46797)
0808 02:25:00.037586 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.037593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.038445 (+ 852us) server_negotiation.cc:407] Connection header received
0808 02:25:00.038666 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.038672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.038777 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.038882 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:00.040015 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.041002 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.042896 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.043592 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.043743 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.047485 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:00.047491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:00.047496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:00.047567 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:00.051021 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.051943 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.051950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.051954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.052160 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.052592 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.052599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.052603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.053163 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:00.053349 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.053960 (+ 611us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.054230 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260808 02:25:00.055444 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.031270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.19.231.61:46471)
0808 02:25:00.032137 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.032151 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.032196 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:25:00.033976 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.034001 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.034447 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.034950 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:00.036855 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.038977 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.040591 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.041378 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.041568 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.046380 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:00.046387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:00.046399 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:00.046431 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:00.050480 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.051236 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.051247 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.051249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.051421 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.051955 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.051962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.051966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.052865 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:00.053222 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.053573 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.054083 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":408,"threads_started":1}
I20260808 02:25:00.055472 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.037328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:34968)
0808 02:25:00.038250 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:00.038300 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:00.038477 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:00.038964 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:00.038971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:00.039609 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:00.039855 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.039864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.041225 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.041235 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.042753 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.042766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.043810 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.043815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.043981 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.044575 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:00.044596 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:00.047278 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:00.051167 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.051180 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.051191 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.051740 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.052315 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.052318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.052320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.052458 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.053332 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:00.053340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:00.053571 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 02:25:00.053805 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 02:25:00.054225 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":92,"thread_start_us":486,"threads_started":1}
I20260808 02:25:00.057698 27425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:25:00 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155899, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.61@KRBTEST.COM for oryx/127.19.231.61@KRBTEST.COM
I20260808 02:25:00.071656 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.058420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.0.0.1:57310)
0808 02:25:00.058660 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:00.058681 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:00.058783 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:00.059114 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:00.059117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:00.059470 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:00.059598 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.059606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.060607 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.060612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.061668 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.061683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.062417 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.062422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.062719 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.063298 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:00.063318 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:00.065900 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:00.068748 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.068755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.068761 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.069314 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.069800 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.069808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.069812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.070030 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.070601 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:00.070606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:00.070803 (+ 197us) client_negotiation.cc:769] Sending connection context
0808 02:25:00.070951 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 02:25:00.071274 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:25:00.071697 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.058706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.19.231.61:46471)
0808 02:25:00.058864 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.058869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.058886 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:25:00.058928 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.058932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.058990 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.059083 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:00.059756 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.060434 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.061837 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.062256 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.062361 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.066068 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:00.066074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:00.066077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:00.066109 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:00.068591 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.069445 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.069453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.069459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.069653 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.070169 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.070174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.070177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.070481 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:00.070676 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.071026 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.071408 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":71}
I20260808 02:25:00.076809 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Bootstrap starting.
I20260808 02:25:00.080099 27425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:00.081188 27425 log.cc:824] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:00.084179 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: No bootstrap required, opened a new log
I20260808 02:25:00.091341 27425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } }
I20260808 02:25:00.091707 27425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:00.091812 27425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7343c0d9e76b42198b991afcbeca6199, State: Initialized, Role: FOLLOWER
I20260808 02:25:00.092501 27425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } }
I20260808 02:25:00.093420 27432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } } }
I20260808 02:25:00.093817 27432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:00.094441 27425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:25:00.104161 27444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:25:00.104319 27444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:25:00.106096 27355 master_runner.cc:428] Detected that this master 7343c0d9e76b42198b991afcbeca6199 is joining an existing cluster
I20260808 02:25:00.106211 27355 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.231.61:46471
I20260808 02:25:00.130127 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.110377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.19.231.62:46797)
0808 02:25:00.110568 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.110576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.111300 (+ 724us) server_negotiation.cc:407] Connection header received
0808 02:25:00.111601 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.111607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.111740 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.111849 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:00.114570 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.116056 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.118808 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.119735 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.119874 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.123108 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:00.123115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:00.123122 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:00.123184 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:00.126886 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.127683 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.127689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.127694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.127861 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.128320 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.128325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.128329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.128705 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:00.128838 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.129469 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.129714 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:25:00.131352 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.110431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:34980)
0808 02:25:00.111168 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:00.111190 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:00.111413 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:00.112843 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:00.112850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:00.113801 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:00.114351 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.114369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.116300 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.116309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.118575 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.118598 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.120261 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.120270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.120512 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.121407 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:00.121440 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:00.122875 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:00.127037 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.127042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.127046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.127528 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.127994 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.128000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.128009 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.128181 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.128922 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:00.128929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:00.129134 (+ 205us) client_negotiation.cc:769] Sending connection context
0808 02:25:00.130428 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0808 02:25:00.130797 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":401,"threads_started":1}
I20260808 02:25:00.138396 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.111617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.19.231.61:46471)
0808 02:25:00.112229 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.112238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.123016 (+ 10778us) server_negotiation.cc:407] Connection header received
0808 02:25:00.123209 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.123217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.123287 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.123361 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:00.124771 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.125946 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.127923 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.128639 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.128755 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.131418 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:00.131423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:00.131427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:00.131465 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:00.135275 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.136146 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.136152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.136155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.136302 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.136769 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.136773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.136776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.137169 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:00.137359 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.137821 (+ 462us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.138038 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427}
I20260808 02:25:00.139643 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.111321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.0.0.1:57312)
0808 02:25:00.122930 (+ 11609us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:00.122951 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:00.123098 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:00.123546 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:00.123550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:00.124245 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:00.124610 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.124623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.126170 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.126174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.127784 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.127800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.129037 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.129048 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.129286 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.129914 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:00.129941 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:00.131203 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:00.135425 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.135429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.135435 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.136021 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.136443 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.136447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.136450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.136623 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.137436 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:00.137440 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:00.137574 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 02:25:00.138673 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0808 02:25:00.139166 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11481,"thread_start_us":11308,"threads_started":1}
WARNING: no policy specified for oryx/127.19.231.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.60@KRBTEST.COM" created.
Aug 08 02:25:00 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.62@KRBTEST.COM for oryx/127.19.231.61@KRBTEST.COM
I20260808 02:25:00.158016 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.138833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.19.231.61:46471)
0808 02:25:00.138991 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.138996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.140216 (+ 1220us) server_negotiation.cc:407] Connection header received
0808 02:25:00.140578 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.140583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.140675 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.140743 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:00.142291 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.143095 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.144473 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.145026 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.145163 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.150173 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:00.150181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:00.150188 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:00.150240 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:00.154263 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.155343 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.155349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.155353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.155562 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.156058 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.156064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.156067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.156455 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:00.156635 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.157271 (+ 636us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.157571 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:25:00.158080 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.139011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.0.0.1:57320)
0808 02:25:00.140078 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:00.140124 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:00.140353 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:00.141031 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:00.141038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:00.141844 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:00.142124 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.142133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.143232 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.143238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.144320 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.144333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.145199 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.145207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.145396 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.146729 (+ 1333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:00.146748 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:00.149896 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:00.154463 (+ 4567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.154483 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.154494 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.155191 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.155710 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.155718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.155724 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.155922 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.156624 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:00.156636 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:00.156890 (+ 254us) client_negotiation.cc:769] Sending connection context
0808 02:25:00.157113 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 02:25:00.157539 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":329,"threads_started":1}
I20260808 02:25:00.160270 26874 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.231.61:46471: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7343c0d9e76b42198b991afcbeca6199"
member_type: NON_VOTER
last_known_addr {
host: "127.19.231.61"
port: 46471
}
attrs {
promote: true
}
}
dest_uuid: "63275c4186764355a53e17d49746ffe9"
cas_config_opid_index: -1
I20260808 02:25:00.161504 26874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } }
I20260808 02:25:00.171914 27457 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7343c0d9e76b42198b991afcbeca6199 (127.19.231.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:00.172748 27400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:00.176679 27400 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Refusing update from remote peer 63275c4186764355a53e17d49746ffe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 02:25:00.177220 27457 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.231.61:46471
I20260808 02:25:00.178337 27355 master.cc:537] Master@127.19.231.61:46471 shutting down...
I20260808 02:25:00.180336 27458 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:00.180409 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:00.180820 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
W20260808 02:25:00.186933 26859 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 02:25:00.189002 26859 consensus_peers.cc:487] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 -> Peer 7343c0d9e76b42198b991afcbeca6199 (127.19.231.61:46471): Couldn't send request to peer 7343c0d9e76b42198b991afcbeca6199. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.60.keytab.
Entry for principal oryx/127.19.231.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.60.keytab.
I20260808 02:25:00.192030 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:34721
--webserver_interface=127.19.231.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.60.keytab
--principal=oryx/127.19.231.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:00.192857 27355 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:00.193326 27355 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:00.193545 27355 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: stopping tablet replica
I20260808 02:25:00.200897 27355 master.cc:552] Master@127.19.231.61:46471 shutdown complete.
I20260808 02:25:00.201061 27355 master_runner.cc:305] Clearing existing system tablet
I20260808 02:25:00.201563 27355 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:00.201670 27355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:00.204649 27355 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:25:00.206266 27463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:00.206786 27355 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:00.207046 27355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:25:00.207139 27355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
uuid: "7343c0d9e76b42198b991afcbeca6199"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "5d21edc4cc0c553b65043259d04af6c9"
server_key_iv: "cf1cba1e2f3785496a9baa4f78457f12"
server_key_version: "encryptionkey@0"
I20260808 02:25:00.210661 27355 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:00.216061 27355 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:25:00.216261 27355 log.cc:1196] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 02:25:00.216668 27355 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Deleting consensus metadata
I20260808 02:25:00.216936 27355 master_runner.cc:315] Copying system tablet from 127.19.231.62:46797
I20260808 02:25:00.225716 27355 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.62:46797
I20260808 02:25:00.243594 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.226635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.19.231.62:46797)
0808 02:25:00.226860 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.226870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.231805 (+ 4935us) server_negotiation.cc:407] Connection header received
0808 02:25:00.232045 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.232049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.232117 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.232228 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:00.233113 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.233984 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.235331 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.235841 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.235971 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.237928 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:00.237939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:00.237942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:00.237974 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:00.240846 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.241495 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.241502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.241505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.241700 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:00.242073 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:00.242096 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:00.242098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:00.242385 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:00.242574 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.242995 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.243202 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:25:00.243691 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.226579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:34982)
0808 02:25:00.231660 (+ 5081us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:00.231679 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:00.231882 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:00.232255 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:00.232277 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:00.232706 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:00.232933 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.232942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.234135 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.234141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.235178 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.235190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.235995 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.236001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.236235 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.236983 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:00.237000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:00.237715 (+ 715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:00.240984 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.240989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.240993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.241367 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.241800 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:00.241805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:00.241809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:00.241956 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:00.242538 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:00.242543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:00.242752 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 02:25:00.242903 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 02:25:00.243237 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4950,"thread_start_us":4802,"threads_started":1}
I20260808 02:25:00.245040 26895 tablet_copy_service.cc:140] P 63275c4186764355a53e17d49746ffe9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7343c0d9e76b42198b991afcbeca6199 ({username='oryx', principal='oryx/127.19.231.61@KRBTEST.COM'} at 127.0.0.1:34982)
I20260808 02:25:00.245455 26895 tablet_copy_service.cc:161] P 63275c4186764355a53e17d49746ffe9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7343c0d9e76b42198b991afcbeca6199 at {username='oryx', principal='oryx/127.19.231.61@KRBTEST.COM'} at 127.0.0.1:34982: session id = 7343c0d9e76b42198b991afcbeca6199-00000000000000000000000000000000
I20260808 02:25:00.249171 26895 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:25:00.251628 27355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:25:00.256330 27355 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: tablet copy: Starting download of 0 data blocks...
I20260808 02:25:00.256708 27355 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: tablet copy: Starting download of 1 WAL segments...
I20260808 02:25:00.259780 27355 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:25:00.262985 26895 tablet_copy_service.cc:342] P 63275c4186764355a53e17d49746ffe9: Request end of tablet copy session 7343c0d9e76b42198b991afcbeca6199-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.231.61@KRBTEST.COM'} at 127.0.0.1:34982
I20260808 02:25:00.263182 26895 tablet_copy_service.cc:433] P 63275c4186764355a53e17d49746ffe9: ending tablet copy session 7343c0d9e76b42198b991afcbeca6199-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7343c0d9e76b42198b991afcbeca6199
I20260808 02:25:00.277014 27355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:00.280357 27470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:00.280779 27471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:00.288478 27473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:00.291045 27355 server_base.cc:1034] running on GCE node
I20260808 02:25:00.291533 27355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:00.295188 27355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:00.296451 27355 hybrid_clock.cc:648] HybridClock initialized: now 1786155900296356 us; error 47 us; skew 500 ppm
I20260808 02:25:00.296711 27355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:00 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155900, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:00.299340 27355 init.cc:377] Logged in from keytab as oryx/127.19.231.61@KRBTEST.COM (short username oryx)
I20260808 02:25:00.302937 27355 webserver.cc:460] Webserver started at http://127.19.231.61:44635/ using document root <none> and password file <none>
I20260808 02:25:00.303436 27355 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:00.303550 27355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:00.307471 27355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260808 02:25:00.309612 27480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:00.310290 27355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:25:00.310451 27355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
uuid: "7343c0d9e76b42198b991afcbeca6199"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "5d21edc4cc0c553b65043259d04af6c9"
server_key_iv: "cf1cba1e2f3785496a9baa4f78457f12"
server_key_version: "encryptionkey@0"
I20260808 02:25:00.310668 27355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:00.330891 27355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:00.335726 27355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:00.365479 27355 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.61:46471
I20260808 02:25:00.365510 27531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.61:46471 every 8 connection(s)
I20260808 02:25:00.366708 27355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
I20260808 02:25:00.372303 27532 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:25:00.376302 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Bootstrap starting.
I20260808 02:25:00.394550 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:00.395118 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Bootstrap complete.
I20260808 02:25:00.396366 27532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } }
I20260808 02:25:00.396695 27532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7343c0d9e76b42198b991afcbeca6199, State: Initialized, Role: LEARNER
I20260808 02:25:00.396939 27532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } }
I20260808 02:25:00.405117 27532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:00.405036 27535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:00.405758 27535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: LEARNER
I20260808 02:25:00.428232 27546 catalog_manager.cc:1259] Loaded cluster ID: d25a7a1b302a4210b8d52e91de56df63
I20260808 02:25:00.428440 27546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: loading cluster ID for follower catalog manager: success
I20260808 02:25:00.432161 27546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: acquiring CA information for follower catalog manager: success
I20260808 02:25:00.435758 27546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:25:00.440291 27349 heartbeater.cc:503] Master 127.19.231.62:46797 was elected leader, sending a full tablet report...
I20260808 02:25:00.659633 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.642401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.0.0.1:57326)
0808 02:25:00.642844 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:00.642866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:00.642999 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:00.648923 (+ 5924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:00.648930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:00.648982 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:00.649417 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.649437 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.652846 (+ 3409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.652854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.655633 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:00.655666 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.658447 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.658473 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:00.658825 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.658919 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 02:25:00.659040 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 02:25:00.659251 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260808 02:25:00.660162 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:00.643541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.19.231.61:46471)
0808 02:25:00.648158 (+ 4617us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:00.648165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:00.648320 (+ 155us) server_negotiation.cc:407] Connection header received
0808 02:25:00.648433 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:00.648440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:00.648569 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:00.648826 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:00.649664 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.652581 (+ 2917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.655929 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:00.658117 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:00.658411 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:00.658934 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:00.659124 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 02:25:00.659355 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4308,"thread_start_us":3954,"threads_started":1}
I20260808 02:25:00.662889 27507 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260808 02:25:00.664177 27535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63275c4186764355a53e17d49746ffe9. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:00.664619 27535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: LEARNER
W20260808 02:25:00.847767 27461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:00.848271 27461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:00.848470 27461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:00.857518 27461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:00.857653 27461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:00.857733 27461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:00.857865 27461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:00.857955 27461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:00.871994 27461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.60.keytab
--principal=oryx/127.19.231.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:34721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.231.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:00.873857 27461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:00.875543 27461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:00.893252 27555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:00.893453 27553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:00.893945 27552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:00.895231 27461 server_base.cc:1034] running on GCE node
I20260808 02:25:00.896067 27461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:00.897689 27461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:00.898932 27461 hybrid_clock.cc:648] HybridClock initialized: now 1786155900898862 us; error 71 us; skew 500 ppm
I20260808 02:25:00.899349 27461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:00 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155900, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:00.903450 27461 init.cc:377] Logged in from keytab as oryx/127.19.231.60@KRBTEST.COM (short username oryx)
I20260808 02:25:00.905471 27461 webserver.cc:460] Webserver started at http://127.19.231.60:42715/ using document root <none> and password file <none>
I20260808 02:25:00.906589 27461 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:00.906816 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:00.907236 27461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:00.911212 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/instance:
uuid: "7344bd2a30da48979061abd3c6d4d7cf"
format_stamp: "Formatted at 2026-08-08 02:25:00 on dist-test-slave-pgkr"
server_key: "0294209da804566a67bd9932434a927b"
server_key_iv: "be5c9d8df64b279601ba03edd6673193"
server_key_version: "encryptionkey@0"
I20260808 02:25:00.912309 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal/instance:
uuid: "7344bd2a30da48979061abd3c6d4d7cf"
format_stamp: "Formatted at 2026-08-08 02:25:00 on dist-test-slave-pgkr"
server_key: "0294209da804566a67bd9932434a927b"
server_key_iv: "be5c9d8df64b279601ba03edd6673193"
server_key_version: "encryptionkey@0"
I20260808 02:25:00.918555 27461 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 02:25:00.922957 27562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:00.924510 27461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:25:00.924790 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
uuid: "7344bd2a30da48979061abd3c6d4d7cf"
format_stamp: "Formatted at 2026-08-08 02:25:00 on dist-test-slave-pgkr"
server_key: "0294209da804566a67bd9932434a927b"
server_key_iv: "be5c9d8df64b279601ba03edd6673193"
server_key_version: "encryptionkey@0"
I20260808 02:25:00.925050 27461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:00.945438 27461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:00.952193 27461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:00.952693 27461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:00.995337 27461 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.60:34721
I20260808 02:25:00.995496 27617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.60:34721 every 8 connection(s)
I20260808 02:25:00.997459 27461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
I20260808 02:25:00.999029 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 27461
I20260808 02:25:01.005117 27619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:01.015523 27619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.60@KRBTEST.COM
I20260808 02:25:01.030958 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.001936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.19.231.60:34721)
0808 02:25:01.003584 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.003603 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.004024 (+ 421us) server_negotiation.cc:407] Connection header received
0808 02:25:01.006683 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.006716 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.007120 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.007620 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.009457 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.011494 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.013313 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.014140 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.014381 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.021179 (+ 6798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.021195 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.021221 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.021283 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.025528 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.026406 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.026422 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.026430 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.026722 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.027199 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.027204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.027208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.027789 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.028185 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.028763 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.029251 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1037,"threads_started":1}
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155900, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.60@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:01.035362 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:01.035686 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.039376 27456 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: attempting to promote NON_VOTER 7343c0d9e76b42198b991afcbeca6199 to VOTER
I20260808 02:25:01.040817 27456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } }
I20260808 02:25:01.043471 27507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 LEARNER]: Refusing update from remote peer 63275c4186764355a53e17d49746ffe9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 02:25:01.043623 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.019280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:34984)
0808 02:25:01.020971 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.021050 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.021264 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.022089 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.022097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.023317 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.023842 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.023853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.025285 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.025299 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.026925 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.026939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.028026 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.028038 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.028284 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.029243 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.029272 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.032882 (+ 3610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.039293 (+ 6411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.039312 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.039322 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.040358 (+ 1036us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.040928 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.040934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.040938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.041309 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.042124 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.042132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.042483 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.042812 (+ 329us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.043142 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1368,"thread_start_us":1114,"threads_started":1}
I20260808 02:25:01.044373 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.019067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.19.231.62:46797)
0808 02:25:01.021755 (+ 2688us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.021760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.021777 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:25:01.021829 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.021833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.021894 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.022032 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.024028 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.025090 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.027108 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.027801 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.027977 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.033098 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.033103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.033106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.033141 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.039109 (+ 5968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.040546 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.040549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.040552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.040780 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.041445 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.041448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.041450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.041896 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.042044 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.042991 (+ 947us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.043339 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2569,"thread_start_us":2465,"threads_started":1}
I20260808 02:25:01.044328 27626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 02:25:01.046032 27619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:25:01.049861 27456 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7343c0d9e76b42198b991afcbeca6199 (127.19.231.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } }
I20260808 02:25:01.050385 27507 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7343c0d9e76b42198b991afcbeca6199 (127.19.231.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } }
I20260808 02:25:01.055164 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } }
I20260808 02:25:01.055487 27535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } }
I20260808 02:25:01.055652 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.055799 27535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:01.056000 27626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } }
I20260808 02:25:01.056529 27626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155900, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.60@KRBTEST.COM for oryx/127.19.231.61@KRBTEST.COM
I20260808 02:25:01.064459 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.046767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.0.0.1:57334)
0808 02:25:01.046968 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.047009 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.047109 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.047518 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.047523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.048068 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.048274 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.048283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.049971 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.049979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.051510 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.051528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.052878 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.052885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.053102 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.053989 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.054012 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.057712 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.061712 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.061716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.061719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.062264 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.062757 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.062760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.062763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.062918 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.063483 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.063487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.063626 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.063766 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.064017 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:25:01.064556 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.046938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.19.231.61:46471)
0808 02:25:01.047127 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.047133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.047185 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:25:01.047240 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.047246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.047330 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.047442 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.048528 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.049807 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.051695 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.052661 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.052810 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.057946 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.057951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.057954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.057987 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.061566 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.062385 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.062389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.062392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.062571 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.063025 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.063029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.063031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.063378 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.063496 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.063924 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.064200 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:25:01.065778 27619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155900, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.60@KRBTEST.COM for oryx/127.19.231.60@KRBTEST.COM
I20260808 02:25:01.081404 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.066491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.0.0.1:36516)
0808 02:25:01.066992 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.067014 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.067147 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.067514 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.067520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.068148 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.068387 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.068396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.069542 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.069548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.070722 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.070735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.071541 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.071548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.071768 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.072373 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.072391 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.075121 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.078795 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.078799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.078802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.079352 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.079730 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.079734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.079736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.079985 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.080561 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.080565 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.080717 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.080829 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.081020 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 02:25:01.081499 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.067097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.19.231.60:34721)
0808 02:25:01.067260 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.067264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.067278 (+ 14us) server_negotiation.cc:407] Connection header received
0808 02:25:01.067317 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.067320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.067371 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.067446 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.068511 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.069400 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.070850 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.071373 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.071469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.075280 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.075284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.075288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.075316 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.078466 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.079466 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.079470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.079472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.079626 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.080090 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.080093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.080095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.080452 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.080560 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.080987 (+ 427us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.081217 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:25:01.088558 27619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Bootstrap starting.
I20260808 02:25:01.093142 27619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:01.094779 27619 log.cc:824] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:01.098349 27619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: No bootstrap required, opened a new log
I20260808 02:25:01.109164 27619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } }
I20260808 02:25:01.109597 27619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:01.109848 27619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7344bd2a30da48979061abd3c6d4d7cf, State: Initialized, Role: FOLLOWER
I20260808 02:25:01.110724 27619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } }
I20260808 02:25:01.112272 27630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } } }
I20260808 02:25:01.112969 27630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:01.113634 27619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:25:01.129163 27641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:25:01.129506 27641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:25:01.132367 27461 master_runner.cc:428] Detected that this master 7344bd2a30da48979061abd3c6d4d7cf is joining an existing cluster
I20260808 02:25:01.132551 27461 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.231.60:34721
I20260808 02:25:01.138005 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26938
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.1@KRBTEST.COM
I20260808 02:25:01.171247 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.139009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.19.231.62:46797)
0808 02:25:01.139247 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.139255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.141093 (+ 1838us) server_negotiation.cc:407] Connection header received
0808 02:25:01.141425 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.141431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.141555 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.141642 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.144030 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.145683 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.150784 (+ 5101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.151917 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.152056 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.161049 (+ 8993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.161058 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.161063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.161170 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.166000 (+ 4830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.166965 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.166971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.166975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.167162 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.167738 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.167743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.167747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.168260 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.168442 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.170701 (+ 2259us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.170928 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 02:25:01.172683 27647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.139420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:34998)
0808 02:25:01.140984 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.141006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.141255 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.141877 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.141885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.143162 (+ 1277us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.143809 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.143827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.146043 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.146052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.150524 (+ 4472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.150543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.154683 (+ 4140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.154697 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.155027 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.156122 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.156152 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.160766 (+ 4614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.166195 (+ 5429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.166201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.166205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.166790 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.167319 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.167324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.167327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.167578 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.168527 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.168548 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.169609 (+ 1061us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.169889 (+ 280us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.170342 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"thread_start_us":932,"threads_started":1}
I20260808 02:25:01.181881 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.140096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.19.231.1:42421)
0808 02:25:01.149131 (+ 9035us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.149143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.150857 (+ 1714us) server_negotiation.cc:407] Connection header received
0808 02:25:01.150979 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.150998 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.151461 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.154692 (+ 3231us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.155214 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.157432 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.159503 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.160718 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.168708 (+ 7990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.169166 (+ 458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.169193 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.169223 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.169272 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.175100 (+ 5828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.176266 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.176275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.176279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.176476 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.177772 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.177777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.177782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.178265 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.180498 (+ 2233us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.180738 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.181070 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8765,"thread_start_us":15824,"threads_started":1}
I20260808 02:25:01.183859 27068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:01.191875 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.141253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.19.231.61:46471)
0808 02:25:01.141452 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.141458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.159804 (+ 18346us) server_negotiation.cc:407] Connection header received
0808 02:25:01.160518 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.160524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.160651 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.160732 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.166532 (+ 5800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.167970 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.174568 (+ 6598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.175628 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.175740 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.179178 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.179185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.179190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.179232 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.184313 (+ 5081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.185520 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.185529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.185535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.185986 (+ 451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.186953 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.186977 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.186984 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.187538 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.187670 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.190883 (+ 3213us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.191282 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 02:25:01.195415 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.141601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.0.0.1:57350)
0808 02:25:01.159683 (+ 18082us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.159704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.160006 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.160986 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.160991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.162285 (+ 1294us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.165550 (+ 3265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.165566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.168239 (+ 2673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.168246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.174323 (+ 6077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.174345 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.176187 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.176196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.176438 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.177264 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.177291 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.178944 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.184504 (+ 5560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.184512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.184518 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.185275 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.186252 (+ 977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.186276 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.186283 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.186710 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.187764 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.187768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.190527 (+ 2759us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.192167 (+ 1640us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.192618 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17859,"mutex_wait_us":2429,"thread_start_us":7857,"threads_started":1}
W20260808 02:25:01.206313 27568 tablet.cc:2369] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.62@KRBTEST.COM for oryx/127.19.231.60@KRBTEST.COM
I20260808 02:25:01.213148 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.150257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.19.231.60:34721)
0808 02:25:01.150500 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.150512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.160400 (+ 9888us) server_negotiation.cc:407] Connection header received
0808 02:25:01.162763 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.162770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.162842 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.162916 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.168679 (+ 5763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.170517 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.191366 (+ 20849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.192773 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.192923 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.201143 (+ 8220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.201152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.201161 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.201197 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.206756 (+ 5559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.209674 (+ 2918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.209681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.209686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.209931 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.210431 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.210440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.210444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.211250 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.211440 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.212455 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.212744 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:25:01.213889 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.149829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.0.0.1:36522)
0808 02:25:01.160313 (+ 10484us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.160331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.162588 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.163123 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.163127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.165169 (+ 2042us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.166266 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.166284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.171672 (+ 5388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.171679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.191208 (+ 19529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.191224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.193496 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.193506 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.193720 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.195516 (+ 1796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.195702 (+ 186us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.200844 (+ 5142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.208839 (+ 7995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.208847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.208854 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.209545 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.210081 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.210085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.210089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.210311 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.211513 (+ 1202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.211517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.211839 (+ 322us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.212701 (+ 862us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.213079 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10331,"thread_start_us":9244,"threads_started":1}
I20260808 02:25:01.221033 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.179775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.0.0.1:36524)
0808 02:25:01.188514 (+ 8739us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.188534 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.188654 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.193455 (+ 4801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.193462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.194373 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.194758 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.194776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.201199 (+ 6423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.201206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.203416 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.203435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.204778 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.204787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.204983 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.207134 (+ 2151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.207165 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.210523 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.217734 (+ 7211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.217740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.217744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.218229 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.218722 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.218726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.218735 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.218926 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.219593 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.219598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.219807 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.219960 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.220362 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8607,"thread_start_us":8393,"threads_started":1}
I20260808 02:25:01.221094 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.180728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.19.231.60:34721)
0808 02:25:01.189404 (+ 8676us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.189410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.189441 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:25:01.189532 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.189537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.189619 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.189720 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.195457 (+ 5737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.201003 (+ 5546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.203641 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.204556 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.205399 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.210953 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.210962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.210967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.210999 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.217522 (+ 6523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.218361 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.218368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.218371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.218541 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.219048 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.219053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.219056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.219453 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.219621 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.220108 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.220332 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8514,"thread_start_us":8296,"threads_started":1}
I20260808 02:25:01.222836 26874 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.231.60:34721: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf"
member_type: NON_VOTER
last_known_addr {
host: "127.19.231.60"
port: 34721
}
attrs {
promote: true
}
}
dest_uuid: "63275c4186764355a53e17d49746ffe9"
cas_config_opid_index: 6
I20260808 02:25:01.223614 26874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } }
I20260808 02:25:01.226431 27507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Refusing update from remote peer 63275c4186764355a53e17d49746ffe9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 02:25:01.227167 27590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:01.228592 27456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 02:25:01.232491 27507 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.231853 27625 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.232870 27590 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 FOLLOWER]: Refusing update from remote peer 63275c4186764355a53e17d49746ffe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 02:25:01.235070 27626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:01.237998 27625 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.231.60:34721
I20260808 02:25:01.238778 27461 master.cc:537] Master@127.19.231.60:34721 shutting down...
I20260808 02:25:01.252447 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.252802 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.252606 27626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.252950 27626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.253123 27535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.253620 27535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:01.258066 27590 raft_consensus.cc:858] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } }
W20260808 02:25:01.258246 27590 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } }
I20260808 02:25:01.280602 27590 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7343c0d9e76b42198b991afcbeca6199 (127.19.231.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:01.284919 27659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:01.285504 27659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 02:25:01.285688 27656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:01.286069 27630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63275c4186764355a53e17d49746ffe9. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: NON_VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: true } } }
I20260808 02:25:01.286319 27630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 02:25:01.286329 27656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 02:25:01.288592 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.288887 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.290258 27461 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260808 02:25:01.290845 27461 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260808 02:25:01.291035 27461 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: stopping tablet replica
W20260808 02:25:01.291349 26859 consensus_peers.cc:487] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 -> Peer 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60:34721): Couldn't send request to peer 7344bd2a30da48979061abd3c6d4d7cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 02:25:01.309132 27461 master.cc:552] Master@127.19.231.60:34721 shutdown complete.
I20260808 02:25:01.309242 27461 master_runner.cc:305] Clearing existing system tablet
I20260808 02:25:01.309648 27461 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:01.309756 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:01.313395 27461 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:25:01.315539 27662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:01.316449 27461 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:01.316726 27461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 02:25:01.316844 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
uuid: "7344bd2a30da48979061abd3c6d4d7cf"
format_stamp: "Formatted at 2026-08-08 02:25:00 on dist-test-slave-pgkr"
server_key: "0294209da804566a67bd9932434a927b"
server_key_iv: "be5c9d8df64b279601ba03edd6673193"
server_key_version: "encryptionkey@0"
I20260808 02:25:01.321969 27461 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:01.325984 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26938
I20260808 02:25:01.331833 27461 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 02:25:01.332039 27461 log.cc:1196] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 02:25:01.332719 27461 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Deleting consensus metadata
I20260808 02:25:01.333017 27461 master_runner.cc:315] Copying system tablet from 127.19.231.62:46797
I20260808 02:25:01.337400 27461 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: tablet copy: Beginning tablet copy session from remote peer at address 127.19.231.62:46797
I20260808 02:25:01.353854 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:42421
--local_ip_for_outbound_sockets=127.19.231.1
--tserver_master_addrs=127.19.231.62:46797
--webserver_port=36317
--webserver_interface=127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab
--principal=oryx/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:01.361547 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.338322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.19.231.62:46797)
0808 02:25:01.338559 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.338565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.345768 (+ 7203us) server_negotiation.cc:407] Connection header received
0808 02:25:01.345947 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.345957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.346079 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.346208 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:01.347406 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.348813 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.350899 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.351649 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.351769 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.354401 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:01.354409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:01.354413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:01.354453 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:01.358165 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.359096 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.359102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.359106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.359268 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:01.359738 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:01.359743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:01.359748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:01.360135 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:01.360312 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.360919 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.361167 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260808 02:25:01.362648 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.338269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:35000)
0808 02:25:01.345600 (+ 7331us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.345629 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.345804 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.346262 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.346268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.346918 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:01.347221 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.347235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.348977 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.348985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.350731 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.350747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.352070 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.352076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.352293 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.353016 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:01.353042 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:01.354170 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:01.358326 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.358331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.358335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.358944 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.359423 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:01.359428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:01.359432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:01.359606 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:01.360404 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:01.360409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:01.360630 (+ 221us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.361826 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.362172 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7168,"thread_start_us":7021,"threads_started":1}
I20260808 02:25:01.363423 26895 tablet_copy_service.cc:140] P 63275c4186764355a53e17d49746ffe9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7344bd2a30da48979061abd3c6d4d7cf ({username='oryx', principal='oryx/127.19.231.60@KRBTEST.COM'} at 127.0.0.1:35000)
I20260808 02:25:01.363606 26895 tablet_copy_service.cc:161] P 63275c4186764355a53e17d49746ffe9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7344bd2a30da48979061abd3c6d4d7cf at {username='oryx', principal='oryx/127.19.231.60@KRBTEST.COM'} at 127.0.0.1:35000: session id = 7344bd2a30da48979061abd3c6d4d7cf-00000000000000000000000000000000
I20260808 02:25:01.367311 26895 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 02:25:01.372321 27461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 02:25:01.379503 27461 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: tablet copy: Starting download of 0 data blocks...
I20260808 02:25:01.379827 27461 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: tablet copy: Starting download of 1 WAL segments...
I20260808 02:25:01.387336 27461 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 02:25:01.392233 26895 tablet_copy_service.cc:342] P 63275c4186764355a53e17d49746ffe9: Request end of tablet copy session 7344bd2a30da48979061abd3c6d4d7cf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.231.60@KRBTEST.COM'} at 127.0.0.1:35000
I20260808 02:25:01.392417 26895 tablet_copy_service.cc:433] P 63275c4186764355a53e17d49746ffe9: ending tablet copy session 7344bd2a30da48979061abd3c6d4d7cf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7344bd2a30da48979061abd3c6d4d7cf
I20260808 02:25:01.405156 27461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:01.408246 27670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:01.408870 27671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:01.418021 27461 server_base.cc:1034] running on GCE node
W20260808 02:25:01.419340 27674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:01.419919 27461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:01.420665 27461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:01.421898 27461 hybrid_clock.cc:648] HybridClock initialized: now 1786155901421853 us; error 30 us; skew 500 ppm
I20260808 02:25:01.422197 27461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155901, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:01.424518 27461 init.cc:377] Logged in from keytab as oryx/127.19.231.60@KRBTEST.COM (short username oryx)
I20260808 02:25:01.427699 27461 webserver.cc:460] Webserver started at http://127.19.231.60:33841/ using document root <none> and password file <none>
I20260808 02:25:01.428087 27461 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:01.428212 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:01.431766 27461 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 02:25:01.439047 27681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:01.439613 27461 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.004s
I20260808 02:25:01.439765 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
uuid: "7344bd2a30da48979061abd3c6d4d7cf"
format_stamp: "Formatted at 2026-08-08 02:25:00 on dist-test-slave-pgkr"
server_key: "0294209da804566a67bd9932434a927b"
server_key_iv: "be5c9d8df64b279601ba03edd6673193"
server_key_version: "encryptionkey@0"
I20260808 02:25:01.439972 27461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:01.462754 27461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:01.467298 27461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:01.500042 27461 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.60:34721
I20260808 02:25:01.500267 27734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.60:34721 every 8 connection(s)
I20260808 02:25:01.501063 27461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
I20260808 02:25:01.507287 27735 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:25:01.511643 27735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Bootstrap starting.
I20260808 02:25:01.530218 27735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:01.531075 27735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: Bootstrap complete.
I20260808 02:25:01.532586 27735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } }
I20260808 02:25:01.532998 27735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7344bd2a30da48979061abd3c6d4d7cf, State: Initialized, Role: LEARNER
I20260808 02:25:01.533380 27735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } }
I20260808 02:25:01.535009 27738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.535416 27735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:01.535485 27738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: LEARNER
I20260808 02:25:01.548110 27749 catalog_manager.cc:1259] Loaded cluster ID: d25a7a1b302a4210b8d52e91de56df63
I20260808 02:25:01.548233 27749 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: loading cluster ID for follower catalog manager: success
I20260808 02:25:01.552331 27749 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: acquiring CA information for follower catalog manager: success
I20260808 02:25:01.555135 27749 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:25:01.760921 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.761221 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.771287 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.761419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.0.0.1:36538)
0808 02:25:01.762115 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:01.762143 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:01.762330 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:01.763425 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:01.763429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:01.763451 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:01.763633 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.763641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.766443 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.766447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.768133 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:01.768147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.770344 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.770349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:01.770566 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.770592 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:25:01.770717 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 02:25:01.770819 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":388,"threads_started":1}
I20260808 02:25:01.771919 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:01.761808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.19.231.60:34721)
0808 02:25:01.762913 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:01.762921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:01.762961 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:25:01.763085 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:01.763099 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:01.763265 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:01.763416 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:01.763797 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.766249 (+ 2452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.768307 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:01.770068 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:01.770275 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:01.770715 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:01.770781 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 02:25:01.770988 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":571,"threads_started":1}
I20260808 02:25:01.773037 27710 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260808 02:25:01.773648 27738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63275c4186764355a53e17d49746ffe9. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: NON_VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: true } } }
I20260808 02:25:01.773972 27738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: LEARNER
I20260808 02:25:01.774431 27456 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: attempting to promote NON_VOTER 7344bd2a30da48979061abd3c6d4d7cf to VOTER
I20260808 02:25:01.775095 27456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } }
I20260808 02:25:01.776743 27710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 LEARNER]: Refusing update from remote peer 63275c4186764355a53e17d49746ffe9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 02:25:01.777261 27507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Refusing update from remote peer 63275c4186764355a53e17d49746ffe9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 02:25:01.777420 27625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 02:25:01.778015 27625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 02:25:01.781185 27625 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.786976 27507 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.788103 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.788450 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.788981 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.789289 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.792440 27753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.792912 27753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:01.794893 27625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.795361 27625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:01.795786 27710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.800084 27738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63275c4186764355a53e17d49746ffe9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:01.800426 27738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:25:01.834075 27669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:01.834450 27669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:01.834707 27669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:01.845443 27669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:01.845598 27669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:01.845911 27669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:25:01.859262 27669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab
--principal=oryx/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:42421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=36317
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:01.860970 27669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:01.862568 27669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:01.881050 27758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:01.882459 27759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:01.882495 27761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:01.884371 27669 server_base.cc:1034] running on GCE node
I20260808 02:25:01.885193 27669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:01.886495 27669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:01.887786 27669 hybrid_clock.cc:648] HybridClock initialized: now 1786155901887708 us; error 78 us; skew 500 ppm
I20260808 02:25:01.888265 27669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:01 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155901, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:01.892654 27669 init.cc:377] Logged in from keytab as oryx/127.19.231.1@KRBTEST.COM (short username oryx)
I20260808 02:25:01.894704 27669 webserver.cc:460] Webserver started at http://127.19.231.1:36317/ using document root <none> and password file <none>
I20260808 02:25:01.895763 27669 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:01.896013 27669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:01.903223 27669 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 02:25:01.907789 27768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:01.909417 27669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:25:01.909566 27669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "92f14ffc1fc14e6ea70b96582bc99e71"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "1bcc9c59dc91b67e6e489d32c6188f90"
server_key_iv: "01f16f0899e93b38bac810e2ec05eb50"
server_key_version: "encryptionkey@0"
I20260808 02:25:01.910888 27669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:01.927860 27669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:01.933604 27669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:01.934051 27669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:01.935700 27669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:01.937881 27669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:01.937973 27669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:01.938050 27669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:01.938107 27669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:02.014021 27669 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:42421
I20260808 02:25:02.014132 27883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:42421 every 8 connection(s)
I20260808 02:25:02.016090 27669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:02.026473 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 27669
I20260808 02:25:02.026976 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 27079
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155901, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.60@KRBTEST.COM
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155901, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155901, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.61@KRBTEST.COM
I20260808 02:25:02.059167 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.021433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:53953 (local address 127.19.231.60:34721)
0808 02:25:02.021704 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.021713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.022866 (+ 1153us) server_negotiation.cc:407] Connection header received
0808 02:25:02.028054 (+ 5188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.028064 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.028186 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.028300 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.032287 (+ 3987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.033944 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.039864 (+ 5920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.041126 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.041279 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.048787 (+ 7508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.048796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.048802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.048848 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.054133 (+ 5285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.055354 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.055362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.055366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.055670 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.056528 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.056535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.056541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.057011 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.057218 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.058382 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.058710 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.2@KRBTEST.COM
I20260808 02:25:02.061120 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.021119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.19.231.1:53953)
0808 02:25:02.022687 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:02.022739 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:02.027663 (+ 4924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:02.028941 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:02.028958 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:02.030207 (+ 1249us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:02.031961 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.032002 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.034212 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.034216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.039544 (+ 5328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.039565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.041953 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.041976 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.042416 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.043484 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:02.043521 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:02.048471 (+ 4950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:02.054329 (+ 5858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.054342 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.054362 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.055190 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.055856 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.055863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.055868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.056348 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.057356 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:02.057367 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:02.057893 (+ 526us) client_negotiation.cc:769] Sending connection context
0808 02:25:02.059439 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0808 02:25:02.059845 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":101,"thread_start_us":668,"threads_started":1}
I20260808 02:25:02.062820 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.023201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:45073 (local address 127.19.231.62:46797)
0808 02:25:02.026343 (+ 3142us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.026353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.026382 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:25:02.028241 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.028247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.028355 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.032333 (+ 3978us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.032603 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.034542 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.039717 (+ 5175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.041120 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.041364 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.050103 (+ 8739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.050115 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.050120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.050164 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.055473 (+ 5309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.056677 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.056682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.056685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.056982 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.057594 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.057598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.057602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.058094 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.058259 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.061150 (+ 2891us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.061945 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2940,"thread_start_us":3191,"threads_started":1}
I20260808 02:25:02.063184 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.023066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.1:45073)
0808 02:25:02.024134 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:02.024152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:02.027671 (+ 3519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:02.028943 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:02.028960 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:02.030957 (+ 1997us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:02.032130 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.032152 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.034871 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.034881 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.039472 (+ 4591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.039494 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.041599 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.041610 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.042338 (+ 728us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.044154 (+ 1816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:02.044190 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:02.049671 (+ 5481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:02.055674 (+ 6003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.055679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.055682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.056514 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.057132 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.057139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.057143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.057452 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.058320 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:02.058326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:02.058504 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 02:25:02.058809 (+ 305us) client_negotiation.cc:240] Negotiation successful
0808 02:25:02.059397 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":266,"thread_start_us":644,"threads_started":1}
I20260808 02:25:02.063685 27884 heartbeater.cc:347] Connected to a master server at 127.19.231.60:34721
I20260808 02:25:02.064072 27885 heartbeater.cc:347] Connected to a master server at 127.19.231.62:46797
I20260808 02:25:02.064244 27885 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:02.064265 27884 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:02.065703 27885 heartbeater.cc:511] Master 127.19.231.62:46797 requested a full tablet report, sending...
I20260808 02:25:02.065702 27884 heartbeater.cc:511] Master 127.19.231.60:34721 requested a full tablet report, sending...
I20260808 02:25:02.068356 26874 ts_manager.cc:194] Re-registered known tserver with Master: 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421)
I20260808 02:25:02.068892 27699 ts_manager.cc:194] Registered new tserver with Master: 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421)
I20260808 02:25:02.069761 26874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:45073
I20260808 02:25:02.073135 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.024503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:56577 (local address 127.19.231.61:46471)
0808 02:25:02.025574 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.025583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.026641 (+ 1058us) server_negotiation.cc:407] Connection header received
0808 02:25:02.033726 (+ 7085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.033733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.033817 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.033892 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.044520 (+ 10628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.046218 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.050143 (+ 3925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.051176 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.051852 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.060343 (+ 8491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.060352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.060371 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.060424 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.068805 (+ 8381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.070354 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.070362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.070366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.070633 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.071340 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.071345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.071349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.071661 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.071821 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.072410 (+ 589us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.072639 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":603,"threads_started":1}
I20260808 02:25:02.073613 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.028353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.19.231.2:36681)
0808 02:25:02.042390 (+ 14037us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.042400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.042434 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:25:02.043050 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.043060 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.043387 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.043573 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.046713 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.049148 (+ 2435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.051954 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.053444 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.053590 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.062847 (+ 9257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.062872 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.062893 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.062945 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.068789 (+ 5844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.070318 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.070329 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.070334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.070532 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.071141 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.071148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.071155 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.071815 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.072272 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.072562 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.072875 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13802,"thread_start_us":14055,"threads_started":1}
I20260808 02:25:02.074128 27892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.025354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.19.231.1:56577)
0808 02:25:02.026516 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:02.026534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:02.033371 (+ 6837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:02.034201 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:02.034205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:02.042454 (+ 8249us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:02.044209 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.044227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.046504 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.046555 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.049580 (+ 3025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.049605 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.052434 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.052442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.052744 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.054935 (+ 2191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:02.054963 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:02.059981 (+ 5018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:02.069137 (+ 9156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.069142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.069146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.070197 (+ 1051us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.070793 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.070801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.070805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.071045 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.071802 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:02.071806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:02.072018 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 02:25:02.073353 (+ 1335us) client_negotiation.cc:240] Negotiation successful
0808 02:25:02.073615 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":133,"thread_start_us":830,"threads_started":1}
I20260808 02:25:02.075305 27205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:02.075845 27886 heartbeater.cc:347] Connected to a master server at 127.19.231.61:46471
I20260808 02:25:02.076085 27886 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:02.076756 27886 heartbeater.cc:511] Master 127.19.231.61:46471 requested a full tablet report, sending...
I20260808 02:25:02.078944 27497 ts_manager.cc:194] Registered new tserver with Master: 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421)
I20260808 02:25:02.169909 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 27079
I20260808 02:25:02.191012 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:36681
--local_ip_for_outbound_sockets=127.19.231.2
--tserver_master_addrs=127.19.231.62:46797
--webserver_port=45369
--webserver_interface=127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab
--principal=oryx/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:02.558454 27896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:02.558849 27896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:02.559020 27896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:02.566504 27896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:02.566690 27896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:02.566874 27896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:25:02.576860 27896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab
--principal=oryx/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:36681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=45369
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:02.578287 27896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:02.579813 27896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:02.593329 27902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:02.593730 27901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:02.593353 27904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:02.599503 27896 server_base.cc:1034] running on GCE node
I20260808 02:25:02.600258 27896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:02.601387 27896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:02.602701 27896 hybrid_clock.cc:648] HybridClock initialized: now 1786155902602662 us; error 54 us; skew 500 ppm
I20260808 02:25:02.603046 27896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155902, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:02.606751 27896 init.cc:377] Logged in from keytab as oryx/127.19.231.2@KRBTEST.COM (short username oryx)
I20260808 02:25:02.608670 27896 webserver.cc:460] Webserver started at http://127.19.231.2:45369/ using document root <none> and password file <none>
I20260808 02:25:02.609383 27896 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:02.609501 27896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:02.615398 27896 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:25:02.618278 27911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:02.619797 27896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:25:02.620021 27896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88"
format_stamp: "Formatted at 2026-08-08 02:24:58 on dist-test-slave-pgkr"
server_key: "56d0db5c0d25f6174c4a405cc1cb4be8"
server_key_iv: "029a6d786c7bb45654e347c90dfe8a77"
server_key_version: "encryptionkey@0"
I20260808 02:25:02.621127 27896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:02.641423 27896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:02.646484 27896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:02.646831 27896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:02.647915 27896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:02.649650 27896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:02.649746 27896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:02.649833 27896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:02.649897 27896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:02.686895 27896 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:36681
I20260808 02:25:02.686951 28026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:36681 every 8 connection(s)
I20260808 02:25:02.688676 27896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:02.698596 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 27896
I20260808 02:25:02.699031 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 27217
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.3@KRBTEST.COM
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155902, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.61@KRBTEST.COM
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155902, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
Aug 08 02:25:02 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155902, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.60@KRBTEST.COM
I20260808 02:25:02.741279 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.700697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.19.231.3:36813)
0808 02:25:02.705029 (+ 4332us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.705037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.711301 (+ 6264us) server_negotiation.cc:407] Connection header received
0808 02:25:02.711503 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.711525 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.711957 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.712165 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.714958 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.717101 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.719688 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.720829 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.720942 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.729338 (+ 8396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.729359 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.729379 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.729505 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.736249 (+ 6744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.737543 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.737553 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.737557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.737741 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.738277 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.738294 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.738296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.738778 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.739101 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.740391 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.740702 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4145,"thread_start_us":3962,"threads_started":1}
I20260808 02:25:02.743698 27344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:02.750424 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.713261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:58159 (local address 127.19.231.60:34721)
0808 02:25:02.723551 (+ 10290us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.723559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.724358 (+ 799us) server_negotiation.cc:407] Connection header received
0808 02:25:02.724446 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.724453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.724546 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.724679 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.727583 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.728984 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.732026 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.732899 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.733072 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.739214 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.739220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.739224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.739267 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.745509 (+ 6242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.746804 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.746809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.746812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.747130 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.747813 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.747818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.747829 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.748312 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.748503 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.749472 (+ 969us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.749814 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10154,"thread_start_us":10036,"threads_started":1}
I20260808 02:25:02.751781 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.712295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.2:52019)
0808 02:25:02.713148 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:02.713175 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:02.717120 (+ 3945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:02.726214 (+ 9094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:02.726220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:02.726970 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:02.727795 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.727813 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.729621 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.729638 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.732190 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.732205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.733522 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.733529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.733863 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.734646 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:02.734701 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:02.738248 (+ 3547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:02.744057 (+ 5809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.744063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.744067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.744869 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.745798 (+ 929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.745808 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.745816 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.746143 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.747262 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:02.747323 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:02.748079 (+ 756us) client_negotiation.cc:769] Sending connection context
0808 02:25:02.748793 (+ 714us) client_negotiation.cc:240] Negotiation successful
0808 02:25:02.749704 (+ 911us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":52,"spinlock_wait_cycles":42112,"thread_start_us":391,"threads_started":1}
I20260808 02:25:02.752013 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.691912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:47687 (local address 127.19.231.61:46471)
0808 02:25:02.704367 (+ 12455us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.704379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.711732 (+ 7353us) server_negotiation.cc:407] Connection header received
0808 02:25:02.717896 (+ 6164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.717907 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.718041 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.718156 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.721929 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.723687 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.728296 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.729290 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.729460 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.735738 (+ 6278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.735745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.735753 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.735803 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.739532 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.740583 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.740592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.740597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.740827 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.741587 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.741595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.741599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.744416 (+ 2817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.744599 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.751155 (+ 6556us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.751424 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12244,"thread_start_us":12426,"threads_started":1}
I20260808 02:25:02.752512 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.719382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.19.231.2:58159)
0808 02:25:02.723863 (+ 4481us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:02.723885 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:02.724045 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:02.724939 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:02.724945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:02.726721 (+ 1776us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:02.727317 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.727334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.729237 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.729243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.731806 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.731828 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.733431 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.733440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.733745 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.734696 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:02.734717 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:02.738958 (+ 4241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:02.745713 (+ 6755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.745718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.745722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.746593 (+ 871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.747320 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.747325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.747329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.747651 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.748579 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:02.748584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:02.748876 (+ 292us) client_negotiation.cc:769] Sending connection context
0808 02:25:02.749032 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:25:02.751290 (+ 2258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4315,"mutex_wait_us":116,"spinlock_wait_cycles":1280,"thread_start_us":4742,"threads_started":1}
I20260808 02:25:02.753211 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.712430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:52019 (local address 127.19.231.62:46797)
0808 02:25:02.725867 (+ 13437us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:02.725875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:02.725895 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:25:02.725955 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:02.725961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:02.726035 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:02.726174 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:02.728002 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.729340 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.732404 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.733273 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.734197 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.738526 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:02.738533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:02.738539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:02.738587 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:02.743861 (+ 5274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.745251 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.745260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.745267 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.745531 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:02.746362 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:02.746371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:02.746378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:02.746991 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:02.752288 (+ 5297us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:02.752453 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 02:25:02.752734 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13292,"thread_start_us":13140,"threads_started":1}
I20260808 02:25:02.753309 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:02.692438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.19.231.2:47687)
0808 02:25:02.711474 (+ 19036us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:02.711549 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:02.717654 (+ 6105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:02.718955 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:02.718974 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:02.720212 (+ 1238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:02.721599 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.721639 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.724080 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.724090 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.728042 (+ 3952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:02.728060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:02.730072 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:02.730084 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:02.730567 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:02.731529 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:02.731572 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:02.735350 (+ 3778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:02.739731 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.739740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.739760 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.740397 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.741016 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:02.741024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:02.741029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:02.741331 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:02.750130 (+ 8799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:02.750135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:02.750357 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 02:25:02.752352 (+ 1995us) client_negotiation.cc:240] Negotiation successful
0808 02:25:02.752722 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18595,"thread_start_us":18486,"threads_started":1}
I20260808 02:25:02.756528 28028 heartbeater.cc:347] Connected to a master server at 127.19.231.62:46797
I20260808 02:25:02.757117 28027 heartbeater.cc:347] Connected to a master server at 127.19.231.60:34721
I20260808 02:25:02.757294 28029 heartbeater.cc:347] Connected to a master server at 127.19.231.61:46471
I20260808 02:25:02.757419 28027 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:02.757535 28029 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:02.758409 28028 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:02.760057 28027 heartbeater.cc:511] Master 127.19.231.60:34721 requested a full tablet report, sending...
I20260808 02:25:02.760329 28028 heartbeater.cc:511] Master 127.19.231.62:46797 requested a full tablet report, sending...
I20260808 02:25:02.761085 28029 heartbeater.cc:511] Master 127.19.231.61:46471 requested a full tablet report, sending...
I20260808 02:25:02.762225 26874 ts_manager.cc:194] Re-registered known tserver with Master: fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681)
I20260808 02:25:02.762290 27699 ts_manager.cc:194] Registered new tserver with Master: fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681)
I20260808 02:25:02.762538 27497 ts_manager.cc:194] Registered new tserver with Master: fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681)
I20260808 02:25:02.763726 26874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:52019
I20260808 02:25:02.850960 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 27217
I20260808 02:25:02.873955 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:36813
--local_ip_for_outbound_sockets=127.19.231.3
--tserver_master_addrs=127.19.231.62:46797
--webserver_port=33827
--webserver_interface=127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab
--principal=oryx/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:03.074985 27885 heartbeater.cc:503] Master 127.19.231.62:46797 was elected leader, sending a full tablet report...
W20260808 02:25:03.226413 28041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:03.226841 28041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:03.227072 28041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:03.236732 28041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:03.236918 28041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:03.237139 28041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:25:03.251006 28041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab
--principal=oryx/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:36813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=33827
--tserver_master_addrs=127.19.231.62:46797,127.19.231.61:46471,127.19.231.60:34721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:03.252961 28041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:03.254848 28041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:03.268177 28046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:03.268137 28047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:03.268215 28049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:03.269017 28041 server_base.cc:1034] running on GCE node
I20260808 02:25:03.269675 28041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:03.270598 28041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:03.271888 28041 hybrid_clock.cc:648] HybridClock initialized: now 1786155903271823 us; error 33 us; skew 500 ppm
I20260808 02:25:03.272212 28041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:03 dist-test-slave-pgkr krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155903, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:03.275645 28041 init.cc:377] Logged in from keytab as oryx/127.19.231.3@KRBTEST.COM (short username oryx)
I20260808 02:25:03.277066 28041 webserver.cc:460] Webserver started at http://127.19.231.3:33827/ using document root <none> and password file <none>
I20260808 02:25:03.277704 28041 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:03.277828 28041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:03.283124 28041 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:25:03.285749 28056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:03.287194 28041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:25:03.287370 28041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "a5b93a6b19e943eea6354fc7971e9e78"
format_stamp: "Formatted at 2026-08-08 02:24:59 on dist-test-slave-pgkr"
server_key: "e85d08ec8f264251f6a4ee1f5d5c93c8"
server_key_iv: "7397d90b5e7efc0a3195c79b66b637eb"
server_key_version: "encryptionkey@0"
I20260808 02:25:03.288467 28041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:03.309998 28041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:03.314198 28041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:03.314522 28041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:03.315683 28041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:03.317251 28041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:03.317371 28041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:03.317487 28041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:03.317585 28041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:03.354971 28041 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:36813
I20260808 02:25:03.355410 28173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:36813 every 8 connection(s)
I20260808 02:25:03.356491 28041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:03.360555 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 28041
Aug 08 02:25:03 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155903, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
Aug 08 02:25:03 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155903, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.60@KRBTEST.COM
I20260808 02:25:03.405929 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.371975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.3:37795)
0808 02:25:03.378193 (+ 6218us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.378218 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.378366 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.379416 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.379433 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.380644 (+ 1211us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:03.381780 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.381814 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.383842 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.383850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.386330 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.386345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.387647 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.387658 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.387996 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.388966 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:03.389001 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:03.392576 (+ 3575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:03.402813 (+ 10237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:03.402818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:03.402821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:03.403359 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:03.403932 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:03.403936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:03.403940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:03.404094 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:03.404740 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:03.404745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:03.404886 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.405042 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.405322 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6038,"thread_start_us":5840,"threads_started":1}
I20260808 02:25:03.406703 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.361106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.19.231.3:36471)
0808 02:25:03.371444 (+ 10338us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.371511 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.374870 (+ 3359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.385433 (+ 10563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.385438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.385962 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:03.386270 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.386281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.387818 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.387823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.389418 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.389440 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.390761 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.390768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.390970 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.391835 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:03.391857 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:03.394685 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:03.400421 (+ 5736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:03.400432 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:03.400451 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:03.401103 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:03.401585 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:03.401591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:03.401594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:03.401879 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:03.402553 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:03.402562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:03.403052 (+ 490us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.403355 (+ 303us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.403937 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9909,"thread_start_us":9619,"threads_started":1}
I20260808 02:25:03.407507 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.372065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:37795 (local address 127.19.231.62:46797)
0808 02:25:03.377485 (+ 5420us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.377494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.378478 (+ 984us) server_negotiation.cc:407] Connection header received
0808 02:25:03.378555 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.378564 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.378710 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.378832 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.382055 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.383605 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.386515 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.387212 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.387348 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.394378 (+ 7030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.394387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.394393 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.394448 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.402667 (+ 8219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.403629 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.403634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.403638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.403793 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.404248 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.404255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.404261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.404598 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.406443 (+ 1845us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.406645 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.406914 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5279,"thread_start_us":5191,"threads_started":1}
I20260808 02:25:03.409389 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.361787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:36471 (local address 127.19.231.60:34721)
0808 02:25:03.362707 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.362718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.378297 (+ 15579us) server_negotiation.cc:407] Connection header received
0808 02:25:03.378387 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.378395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.378515 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.378674 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.386489 (+ 7815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.387607 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.389607 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.390272 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.390395 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.394962 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.394974 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.394983 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.395055 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.400247 (+ 5192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.401269 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.401275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.401278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.401438 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.402037 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.402042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.402046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.402399 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.408521 (+ 6122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.408662 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.408919 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":552,"threads_started":1}
I20260808 02:25:03.410898 28175 heartbeater.cc:347] Connected to a master server at 127.19.231.62:46797
I20260808 02:25:03.411304 28175 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:03.413550 28175 heartbeater.cc:511] Master 127.19.231.62:46797 requested a full tablet report, sending...
Aug 08 02:25:03 dist-test-slave-pgkr krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155903, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.61@KRBTEST.COM
I20260808 02:25:03.417521 26874 ts_manager.cc:194] Re-registered known tserver with Master: a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813)
I20260808 02:25:03.420424 26874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:37795
I20260808 02:25:03.423009 28174 heartbeater.cc:347] Connected to a master server at 127.19.231.60:34721
I20260808 02:25:03.423187 28174 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:03.424010 28174 heartbeater.cc:511] Master 127.19.231.60:34721 requested a full tablet report, sending...
I20260808 02:25:03.426196 27699 ts_manager.cc:194] Registered new tserver with Master: a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813)
I20260808 02:25:03.432615 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.397546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.19.231.61:46471)
0808 02:25:03.398309 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.398317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.399016 (+ 699us) server_negotiation.cc:407] Connection header received
0808 02:25:03.399312 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.399319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.399402 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.399482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.401620 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.403501 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.410151 (+ 6650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.411221 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.411357 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.417616 (+ 6259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.417624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.417629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.417670 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.426711 (+ 9041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.428417 (+ 1706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.428436 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.428440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.428748 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.429263 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.429269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.429273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.429771 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.430078 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.431413 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.431832 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":5257,"threads_started":1}
I20260808 02:25:03.435019 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.391459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.19.231.62:46797)
0808 02:25:03.398104 (+ 6645us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.398111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.398132 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:25:03.398183 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.398188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.398261 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.398374 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.413649 (+ 15275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.415411 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.418263 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.419299 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.419463 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.424867 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.424874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.424881 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.424925 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.430360 (+ 5435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.431946 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.431951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.431955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.432143 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.432820 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.432825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.432828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.433328 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.433606 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.434281 (+ 675us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.434510 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6503,"thread_start_us":6371,"threads_started":1}
I20260808 02:25:03.438468 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.378278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.19.231.3:51257)
0808 02:25:03.393655 (+ 15377us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.393675 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.393812 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.394903 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.394908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.395549 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:03.395957 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.395973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.405591 (+ 9618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.405597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.407882 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.407897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.409850 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.409858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.410084 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.411224 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:03.411246 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:03.424804 (+ 13558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:03.434649 (+ 9845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:03.434655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:03.434659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:03.435294 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:03.435810 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:03.435817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:03.435822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:03.436351 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:03.437150 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:03.437156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:03.437354 (+ 198us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.437504 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.437869 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15237,"thread_start_us":15095,"threads_started":1}
I20260808 02:25:03.438882 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.378380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:51257 (local address 127.19.231.61:46471)
0808 02:25:03.391636 (+ 13256us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.391642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.393791 (+ 2149us) server_negotiation.cc:407] Connection header received
0808 02:25:03.393936 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.393946 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.394093 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.394217 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.403058 (+ 8841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.405194 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.408055 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.409164 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.409301 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.427664 (+ 18363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.427672 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.427677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.427766 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.433584 (+ 5818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.435427 (+ 1843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.435434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.435437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.435646 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.436505 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.436510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.436513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.436985 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.437838 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.438024 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.438284 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13112,"thread_start_us":13604,"threads_started":1}
I20260808 02:25:03.439683 28176 heartbeater.cc:347] Connected to a master server at 127.19.231.61:46471
I20260808 02:25:03.439891 28176 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:03.440526 28176 heartbeater.cc:511] Master 127.19.231.61:46471 requested a full tablet report, sending...
I20260808 02:25:03.442065 27497 ts_manager.cc:194] Registered new tserver with Master: a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813)
I20260808 02:25:03.445422 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.399005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.19.231.60:34721)
0808 02:25:03.410475 (+ 11470us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.410481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.421318 (+ 10837us) server_negotiation.cc:407] Connection header received
0808 02:25:03.421659 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.421666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.421815 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.421910 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.425788 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.427258 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.429835 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.430863 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.430970 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.436675 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.436681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.436686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.436720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.442139 (+ 5419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.443069 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.443073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.443075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.443273 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.443797 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.443801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.443803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.444138 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.444350 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.444914 (+ 564us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.445099 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10507,"thread_start_us":20871,"threads_started":1}
I20260808 02:25:03.440667 26874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:03.451392 26874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:03.494472 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.479281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.19.231.2:36681)
0808 02:25:03.480290 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.480304 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.481782 (+ 1478us) server_negotiation.cc:407] Connection header received
0808 02:25:03.482102 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.482111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.482492 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.482772 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:03.484322 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.486661 (+ 2339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.491495 (+ 4834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.492630 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.492730 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.492884 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.493986 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.494171 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":525,"threads_started":1}
I20260808 02:25:03.496583 28204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.478767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:36681 (local address 127.0.0.1:42378)
0808 02:25:03.481576 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.481593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.481852 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.483092 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.483100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.483137 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:03.484042 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.484059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.486921 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.486930 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.491231 (+ 4301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.491249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.493543 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.493556 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.493757 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.493797 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.493916 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.494062 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2653,"thread_start_us":2501,"threads_started":1}
I20260808 02:25:03.500057 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.481605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42421 (local address 127.0.0.1:32966)
0808 02:25:03.487753 (+ 6148us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.487776 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.487909 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.489291 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.489299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.489324 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:03.490598 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.490614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.493127 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.493136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.496351 (+ 3215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.496366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.499037 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.499056 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.499290 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.499328 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.499446 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.499580 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5951,"thread_start_us":821,"threads_started":1}
I20260808 02:25:03.500173 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.482023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.19.231.1:42421)
0808 02:25:03.488612 (+ 6589us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.488620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.488648 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:25:03.488801 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.488810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.489101 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.489910 (+ 809us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:03.490918 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.492866 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.496664 (+ 3798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.498712 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.498827 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.498988 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.499554 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.499745 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6334,"thread_start_us":6961,"threads_started":1}
I20260808 02:25:03.501832 27961 tablet_service.cc:1467] Processing CreateTablet for tablet beef83d600ba4ddb989611e62f37a2f1 (DEFAULT_TABLE table=test-table [id=8d0ef74137c449738077f4bb64bc1f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:03.503727 27961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beef83d600ba4ddb989611e62f37a2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:03.505028 27818 tablet_service.cc:1467] Processing CreateTablet for tablet beef83d600ba4ddb989611e62f37a2f1 (DEFAULT_TABLE table=test-table [id=8d0ef74137c449738077f4bb64bc1f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:03.507637 27818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beef83d600ba4ddb989611e62f37a2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:03.512040 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.478856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:36813 (local address 127.0.0.1:52534)
0808 02:25:03.482927 (+ 4071us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.482950 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.483115 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.488589 (+ 5474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.488594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.488621 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:03.489793 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.489810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.493167 (+ 3357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.493173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.507817 (+ 14644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.507838 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.510885 (+ 3047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.510893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.511200 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.511238 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.511328 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.511477 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3874,"thread_start_us":8394,"threads_started":1}
I20260808 02:25:03.513248 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.479394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.19.231.3:36813)
0808 02:25:03.481180 (+ 1786us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.481190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.487889 (+ 6699us) server_negotiation.cc:407] Connection header received
0808 02:25:03.488017 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.488027 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.488343 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.488581 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:03.490289 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.492864 (+ 2575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.508069 (+ 15205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.510209 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.510340 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.510549 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.512688 (+ 2139us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.512858 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1558,"thread_start_us":1277,"threads_started":1}
I20260808 02:25:03.517417 28108 tablet_service.cc:1467] Processing CreateTablet for tablet beef83d600ba4ddb989611e62f37a2f1 (DEFAULT_TABLE table=test-table [id=8d0ef74137c449738077f4bb64bc1f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:03.518893 28108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beef83d600ba4ddb989611e62f37a2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:03.526467 28210 tablet_bootstrap.cc:492] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Bootstrap starting.
I20260808 02:25:03.531982 28210 tablet_bootstrap.cc:654] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:03.534351 28210 log.cc:824] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:03.535662 28211 tablet_bootstrap.cc:492] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Bootstrap starting.
I20260808 02:25:03.539111 28213 tablet_bootstrap.cc:492] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Bootstrap starting.
I20260808 02:25:03.540192 28210 tablet_bootstrap.cc:492] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: No bootstrap required, opened a new log
I20260808 02:25:03.540566 28210 ts_tablet_manager.cc:1397] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260808 02:25:03.540853 28211 tablet_bootstrap.cc:654] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:03.542876 28211 log.cc:824] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:03.545320 28213 tablet_bootstrap.cc:654] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:03.546262 28211 tablet_bootstrap.cc:492] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: No bootstrap required, opened a new log
I20260808 02:25:03.546561 28211 ts_tablet_manager.cc:1397] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260808 02:25:03.547277 28213 log.cc:824] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:03.551494 28213 tablet_bootstrap.cc:492] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: No bootstrap required, opened a new log
I20260808 02:25:03.551847 28213 ts_tablet_manager.cc:1397] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260808 02:25:03.556196 28210 raft_consensus.cc:348] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.556960 28210 raft_consensus.cc:374] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:03.557181 28210 raft_consensus.cc:729] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1f9169f4c84130b2fa24fb7d9dcf88, State: Initialized, Role: FOLLOWER
I20260808 02:25:03.557931 28211 raft_consensus.cc:348] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.558466 28211 raft_consensus.cc:374] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:03.558187 28210 consensus_queue.cc:260] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.558706 28211 raft_consensus.cc:729] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f14ffc1fc14e6ea70b96582bc99e71, State: Initialized, Role: FOLLOWER
I20260808 02:25:03.559933 28028 heartbeater.cc:503] Master 127.19.231.62:46797 was elected leader, sending a full tablet report...
I20260808 02:25:03.563733 28211 consensus_queue.cc:260] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.564076 28210 ts_tablet_manager.cc:1428] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
I20260808 02:25:03.568419 28211 ts_tablet_manager.cc:1428] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Time spent starting tablet: real 0.022s user 0.008s sys 0.007s
I20260808 02:25:03.567464 28213 raft_consensus.cc:348] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.569774 28213 raft_consensus.cc:374] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:03.569929 28213 raft_consensus.cc:729] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b93a6b19e943eea6354fc7971e9e78, State: Initialized, Role: FOLLOWER
I20260808 02:25:03.570824 28213 consensus_queue.cc:260] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.573911 28213 ts_tablet_manager.cc:1428] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Time spent starting tablet: real 0.022s user 0.011s sys 0.009s
I20260808 02:25:03.574316 28175 heartbeater.cc:503] Master 127.19.231.62:46797 was elected leader, sending a full tablet report...
I20260808 02:25:03.591176 28218 raft_consensus.cc:482] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:03.591766 28218 raft_consensus.cc:504] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.594019 28218 leader_election.cc:288] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681), 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421)
I20260808 02:25:03.601495 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.595266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:36053 (local address 127.19.231.1:42421)
0808 02:25:03.595437 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.595441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.595456 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:25:03.595532 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.595539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.595618 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.595733 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:03.596072 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.597345 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.599473 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.600482 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.600689 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.600763 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.601089 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.601218 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:25:03.601521 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.594494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:42159 (local address 127.19.231.2:36681)
0808 02:25:03.594807 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.594813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.594844 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:25:03.594947 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.594954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.595049 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.595154 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:03.595588 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.597082 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.599263 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.600361 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.600549 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.600629 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.601046 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.601183 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 02:25:03.601540 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.594164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:36681 (local address 127.19.231.3:42159)
0808 02:25:03.594542 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.594563 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.594715 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.595184 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.595190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.595212 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:03.595451 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.595460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.597235 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.597239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.599075 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.599088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.600584 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.600591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.600843 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.600871 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.601016 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.601156 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2}
I20260808 02:25:03.601543 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.594670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42421 (local address 127.19.231.3:36053)
0808 02:25:03.594915 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:03.594962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:03.595056 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:03.595737 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:03.595740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:03.595760 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:03.595913 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.595919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.597576 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.597582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.599326 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:03.599339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.600689 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.600696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:03.600917 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.600941 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:25:03.601044 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:25:03.601187 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260808 02:25:03.602802 27838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beef83d600ba4ddb989611e62f37a2f1" candidate_uuid: "a5b93a6b19e943eea6354fc7971e9e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" is_pre_election: true
I20260808 02:25:03.602809 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beef83d600ba4ddb989611e62f37a2f1" candidate_uuid: "a5b93a6b19e943eea6354fc7971e9e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" is_pre_election: true
I20260808 02:25:03.603500 27838 raft_consensus.cc:2454] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b93a6b19e943eea6354fc7971e9e78 in term 0.
I20260808 02:25:03.603585 27981 raft_consensus.cc:2454] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b93a6b19e943eea6354fc7971e9e78 in term 0.
I20260808 02:25:03.604461 28060 leader_election.cc:302] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f14ffc1fc14e6ea70b96582bc99e71, a5b93a6b19e943eea6354fc7971e9e78; no voters:
I20260808 02:25:03.604848 28218 raft_consensus.cc:2781] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:03.604996 28218 raft_consensus.cc:482] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:03.605212 28218 raft_consensus.cc:3037] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:03.608421 28218 raft_consensus.cc:504] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:03.609189 28218 leader_election.cc:288] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [CANDIDATE]: Term 1 election: Requested vote from peers fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681), 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421)
I20260808 02:25:03.609714 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beef83d600ba4ddb989611e62f37a2f1" candidate_uuid: "a5b93a6b19e943eea6354fc7971e9e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88"
I20260808 02:25:03.609907 27838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beef83d600ba4ddb989611e62f37a2f1" candidate_uuid: "a5b93a6b19e943eea6354fc7971e9e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f14ffc1fc14e6ea70b96582bc99e71"
I20260808 02:25:03.610033 27981 raft_consensus.cc:3037] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:03.610213 27838 raft_consensus.cc:3037] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:03.613513 27838 raft_consensus.cc:2454] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b93a6b19e943eea6354fc7971e9e78 in term 1.
I20260808 02:25:03.613692 27981 raft_consensus.cc:2454] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b93a6b19e943eea6354fc7971e9e78 in term 1.
I20260808 02:25:03.613981 28060 leader_election.cc:302] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f14ffc1fc14e6ea70b96582bc99e71, a5b93a6b19e943eea6354fc7971e9e78; no voters:
I20260808 02:25:03.614540 28218 raft_consensus.cc:2781] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:03.615459 28218 raft_consensus.cc:686] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 1 LEADER]: Becoming Leader. State: Replica: a5b93a6b19e943eea6354fc7971e9e78, State: Running, Role: LEADER
I20260808 02:25:03.616209 28218 consensus_queue.cc:237] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
W20260808 02:25:03.617496 28177 tablet.cc:2369] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:03.623088 26873 catalog_manager.cc:5495] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "a5b93a6b19e943eea6354fc7971e9e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } health_report { overall_health: HEALTHY } } }
I20260808 02:25:03.661108 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.654170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.19.231.3:36813)
0808 02:25:03.654435 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.654440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.654672 (+ 232us) server_negotiation.cc:407] Connection header received
0808 02:25:03.655082 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.655086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.655228 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.655362 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:03.655808 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.656849 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.658551 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.659149 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.659263 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.659793 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:03.660395 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:03.660511 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.660689 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.660765 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:25:03.671696 27838 raft_consensus.cc:1261] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 FOLLOWER]: Refusing update from remote peer a5b93a6b19e943eea6354fc7971e9e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:03.672017 27981 raft_consensus.cc:1261] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Refusing update from remote peer a5b93a6b19e943eea6354fc7971e9e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:03.672933 28219 consensus_queue.cc:1035] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 02:25:03.673703 28218 consensus_queue.cc:1035] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:25:03.698006 28030 tablet.cc:2369] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:03.702910 28225 mvcc.cc:205] Tried to move back new op lower bound from 7316094581427314688 to 7316094581217300480. Current Snapshot: MvccSnapshot[applied={T|T < 7316094581427314688}]
I20260808 02:25:03.706491 28226 mvcc.cc:205] Tried to move back new op lower bound from 7316094581427314688 to 7316094581217300480. Current Snapshot: MvccSnapshot[applied={T|T < 7316094581427314688}]
I20260808 02:25:03.742192 26874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:03.742820 26874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:03.754179 26874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63275c4186764355a53e17d49746ffe9: Sending DeleteTablet for 3 replicas of tablet beef83d600ba4ddb989611e62f37a2f1
I20260808 02:25:03.755633 27961 tablet_service.cc:1514] Processing DeleteTablet for tablet beef83d600ba4ddb989611e62f37a2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:03 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:42378
I20260808 02:25:03.755956 28108 tablet_service.cc:1514] Processing DeleteTablet for tablet beef83d600ba4ddb989611e62f37a2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:03 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:52534
I20260808 02:25:03.756156 27818 tablet_service.cc:1514] Processing DeleteTablet for tablet beef83d600ba4ddb989611e62f37a2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:03 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:32966
I20260808 02:25:03.767491 28238 tablet_replica.cc:330] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: stopping tablet replica
I20260808 02:25:03.768112 28238 raft_consensus.cc:2229] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:03.768904 28238 raft_consensus.cc:2258] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:03.771870 28238 ts_tablet_manager.cc:1905] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:03.774075 28237 tablet_replica.cc:330] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: stopping tablet replica
I20260808 02:25:03.774302 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 26841
I20260808 02:25:03.774780 28237 raft_consensus.cc:2229] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:03.775905 26907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:03.782219 28237 raft_consensus.cc:2258] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:03.784225 28238 ts_tablet_manager.cc:1918] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:03.784467 28238 log.cc:1196] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/beef83d600ba4ddb989611e62f37a2f1
I20260808 02:25:03.784504 28236 tablet_replica.cc:330] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: stopping tablet replica
I20260808 02:25:03.785246 28236 raft_consensus.cc:2229] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:03.785600 28238 ts_tablet_manager.cc:1939] T beef83d600ba4ddb989611e62f37a2f1 P 92f14ffc1fc14e6ea70b96582bc99e71: Deleting consensus metadata
I20260808 02:25:03.786218 28236 raft_consensus.cc:2258] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:03.789522 28236 ts_tablet_manager.cc:1905] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:03.790287 28237 ts_tablet_manager.cc:1905] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:03.801589 28237 ts_tablet_manager.cc:1918] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:03.802008 28237 log.cc:1196] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/beef83d600ba4ddb989611e62f37a2f1
I20260808 02:25:03.801931 28236 ts_tablet_manager.cc:1918] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:03.802301 28236 log.cc:1196] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/beef83d600ba4ddb989611e62f37a2f1
I20260808 02:25:03.803136 28237 ts_tablet_manager.cc:1939] T beef83d600ba4ddb989611e62f37a2f1 P a5b93a6b19e943eea6354fc7971e9e78: Deleting consensus metadata
I20260808 02:25:03.803409 28236 ts_tablet_manager.cc:1939] T beef83d600ba4ddb989611e62f37a2f1 P fa1f9169f4c84130b2fa24fb7d9dcf88: Deleting consensus metadata
I20260808 02:25:03.820672 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.760731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.19.231.61:46471)
0808 02:25:03.761000 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.761010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.770673 (+ 9663us) server_negotiation.cc:407] Connection header received
0808 02:25:03.778262 (+ 7589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.778268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.778428 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.778520 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.791491 (+ 12971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.794099 (+ 2608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.796788 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.798110 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.798249 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.810212 (+ 11963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.810221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.810227 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.810280 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.816142 (+ 5862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.817404 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.817410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.817414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.817618 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.818284 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.818290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.818293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.818866 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.819168 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.819829 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.820139 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:25:03.848627 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.773377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.19.231.60:34721)
0808 02:25:03.773606 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.773613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.820824 (+ 47211us) server_negotiation.cc:407] Connection header received
0808 02:25:03.821159 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.821166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.821277 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.821382 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.824952 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.826540 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.830493 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.831487 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.831637 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.838474 (+ 6837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.838481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.838487 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.838533 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.844449 (+ 5916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.845936 (+ 1487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.845942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.845946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.846138 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.846718 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.846724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.846728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.847191 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.847571 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.847986 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.848213 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:25:03.933421 26859 catalog_manager.cc:4841] TS fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681): tablet beef83d600ba4ddb989611e62f37a2f1 (table test-table [id=8d0ef74137c449738077f4bb64bc1f4a]) successfully deleted
I20260808 02:25:03.934557 26860 catalog_manager.cc:4841] TS a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813): tablet beef83d600ba4ddb989611e62f37a2f1 (table test-table [id=8d0ef74137c449738077f4bb64bc1f4a]) successfully deleted
I20260808 02:25:03.934628 26859 catalog_manager.cc:4841] TS 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421): tablet beef83d600ba4ddb989611e62f37a2f1 (table test-table [id=8d0ef74137c449738077f4bb64bc1f4a]) successfully deleted
I20260808 02:25:03.935384 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 26841
I20260808 02:25:03.993618 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.976628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.19.231.61:46471)
0808 02:25:03.976883 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.976893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.977043 (+ 150us) server_negotiation.cc:407] Connection header received
0808 02:25:03.977256 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.977265 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.977373 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.977479 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.979724 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.980903 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.982201 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.982888 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.983023 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.986478 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.986484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.986490 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.986539 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.990066 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.991148 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.991154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.991157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.991305 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.991778 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.991785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.991790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.992211 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.992466 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.993006 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.993268 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260808 02:25:03.994017 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:03.977946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.19.231.60:34721)
0808 02:25:03.978118 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:03.978122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:03.978349 (+ 227us) server_negotiation.cc:407] Connection header received
0808 02:25:03.978564 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:03.978570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:03.978692 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:03.978806 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:03.980766 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.981911 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.983455 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:03.984149 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:03.984279 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:03.987278 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:03.987285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:03.987291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:03.987334 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:03.990099 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.991148 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.991153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.991157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.991305 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:03.991732 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:03.991740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:03.991742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:03.992112 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:03.992394 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:03.993290 (+ 896us) server_negotiation.cc:299] Negotiation successful
0808 02:25:03.993659 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 02:25:04.730118 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.729112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.1:45081)
0808 02:25:04.729597 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.729869 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":213,"threads_started":1}
W20260808 02:25:04.730762 27885 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:46797 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:04.731107 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.730243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.2:52323)
0808 02:25:04.730763 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.730926 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"threads_started":1}
W20260808 02:25:04.731864 28028 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:46797 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:04.735631 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:04.734845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.3:47147)
0808 02:25:04.735263 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:04.735376 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":210,"threads_started":1}
W20260808 02:25:04.735991 28175 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:46797 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:05.253443 28251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63275c4186764355a53e17d49746ffe9)
I20260808 02:25:05.253575 28252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63275c4186764355a53e17d49746ffe9)
I20260808 02:25:05.253856 28252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } }
I20260808 02:25:05.253854 28251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } }
I20260808 02:25:05.255257 28252 leader_election.cc:288] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63275c4186764355a53e17d49746ffe9 (127.19.231.62:46797), 7343c0d9e76b42198b991afcbeca6199 (127.19.231.61:46471)
I20260808 02:25:05.255370 28251 leader_election.cc:288] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63275c4186764355a53e17d49746ffe9 (127.19.231.62:46797), 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60:34721)
I20260808 02:25:05.256661 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.255498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:35448)
0808 02:25:05.256095 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.256329 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":250,"threads_started":1}
W20260808 02:25:05.264083 27483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:05.265012 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.255711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:35450)
0808 02:25:05.264427 (+ 8716us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.264629 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8589,"thread_start_us":8339,"threads_started":1}
I20260808 02:25:05.267032 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.256144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:34721 (local address 127.0.0.1:36580)
0808 02:25:05.257051 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.257074 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.257267 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.257785 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.257793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.257827 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.258136 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.258160 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.260434 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.260442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.263349 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.263371 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.265809 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.265819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.266165 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.266222 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.266356 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.266583 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":7550,"threads_started":1}
I20260808 02:25:05.267213 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.256383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.19.231.60:34721)
0808 02:25:05.256988 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.256999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.257188 (+ 189us) server_negotiation.cc:407] Connection header received
0808 02:25:05.257449 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.257459 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.257606 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.257773 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.258370 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.260268 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.263571 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.265468 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.265757 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.265921 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.266435 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.266672 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":273,"threads_started":1}
W20260808 02:25:05.272032 27685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:05.272301 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.271850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:35456)
0808 02:25:05.272016 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.272110 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 02:25:05.272600 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7343c0d9e76b42198b991afcbeca6199" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7344bd2a30da48979061abd3c6d4d7cf" is_pre_election: true
I20260808 02:25:05.273109 27710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7343c0d9e76b42198b991afcbeca6199 in term 1.
W20260808 02:25:05.273141 27483 leader_election.cc:334] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63275c4186764355a53e17d49746ffe9 (127.19.231.62:46797): Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:05.273818 27483 leader_election.cc:302] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7343c0d9e76b42198b991afcbeca6199, 7344bd2a30da48979061abd3c6d4d7cf; no voters: 63275c4186764355a53e17d49746ffe9
I20260808 02:25:05.274236 28251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 02:25:05.274302 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.273708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:35466)
0808 02:25:05.273897 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.274075 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:25:05.274385 28251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 63275c4186764355a53e17d49746ffe9)
I20260808 02:25:05.274528 28251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 1 FOLLOWER]: Advancing to term 2
W20260808 02:25:05.274762 27685 leader_election.cc:334] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63275c4186764355a53e17d49746ffe9 (127.19.231.62:46797): Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:05.277983 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.264594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46471 (local address 127.0.0.1:57390)
0808 02:25:05.265358 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.265381 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.265530 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.272209 (+ 6679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.272214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.272244 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.272610 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.272630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.274657 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.274665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.276160 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.276173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.277450 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.277459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.277609 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.277636 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.277714 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.277807 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":7099,"threads_started":1}
I20260808 02:25:05.278388 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.271098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.19.231.61:46471)
0808 02:25:05.271815 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.271822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.271847 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:25:05.271916 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.271921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.272008 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.272804 (+ 796us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.273091 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.274441 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.276318 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.277226 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.277382 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.277522 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.277766 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.277925 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":380,"threads_started":1}
I20260808 02:25:05.279100 27507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7344bd2a30da48979061abd3c6d4d7cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7343c0d9e76b42198b991afcbeca6199" is_pre_election: true
I20260808 02:25:05.280241 28251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } }
I20260808 02:25:05.280728 27507 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7344bd2a30da48979061abd3c6d4d7cf in current term 2: Already voted for candidate 7343c0d9e76b42198b991afcbeca6199 in this term.
I20260808 02:25:05.281082 28251 leader_election.cc:288] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [CANDIDATE]: Term 2 election: Requested vote from peers 63275c4186764355a53e17d49746ffe9 (127.19.231.62:46797), 7344bd2a30da48979061abd3c6d4d7cf (127.19.231.60:34721)
I20260808 02:25:05.281531 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.281122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:35482)
0808 02:25:05.281283 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.281392 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 02:25:05.281792 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7343c0d9e76b42198b991afcbeca6199" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7344bd2a30da48979061abd3c6d4d7cf"
I20260808 02:25:05.282099 27710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:25:05.282446 27685 leader_election.cc:302] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7344bd2a30da48979061abd3c6d4d7cf; no voters: 63275c4186764355a53e17d49746ffe9, 7343c0d9e76b42198b991afcbeca6199
I20260808 02:25:05.282651 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.282208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:35484)
0808 02:25:05.282363 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.282443 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 02:25:05.284518 27710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7343c0d9e76b42198b991afcbeca6199 in term 2.
I20260808 02:25:05.284726 28252 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 02:25:05.284837 27483 leader_election.cc:334] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 63275c4186764355a53e17d49746ffe9 (127.19.231.62:46797): Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
I20260808 02:25:05.285101 27483 leader_election.cc:302] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7343c0d9e76b42198b991afcbeca6199, 7344bd2a30da48979061abd3c6d4d7cf; no voters: 63275c4186764355a53e17d49746ffe9
I20260808 02:25:05.285370 28251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:25:05.285837 28251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [term 2 LEADER]: Becoming Leader. State: Replica: 7343c0d9e76b42198b991afcbeca6199, State: Running, Role: LEADER
I20260808 02:25:05.286242 28251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } }
I20260808 02:25:05.288498 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7343c0d9e76b42198b991afcbeca6199. Latest consensus state: current_term: 2 leader_uuid: "7343c0d9e76b42198b991afcbeca6199" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:05.288998 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:05.290161 28263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:05.294602 28263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:05.295516 28263 catalog_manager.cc:1259] Loaded cluster ID: d25a7a1b302a4210b8d52e91de56df63
I20260808 02:25:05.295631 28263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:05.296820 28263 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:05.297652 28263 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Loaded TSK: 0
I20260808 02:25:05.298506 28263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:25:05.344816 27497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:05.352149 27497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:05.356729 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.356296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:53608)
0808 02:25:05.356456 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.356576 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 02:25:05.357193 27710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [term 2 FOLLOWER]: Refusing update from remote peer 7343c0d9e76b42198b991afcbeca6199: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 02:25:05.358122 28261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 02:25:05.358682 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.358221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:53616)
0808 02:25:05.358428 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.358497 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 02:25:05.358908 27483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 -> Peer 63275c4186764355a53e17d49746ffe9 (127.19.231.62:46797): Couldn't send request to peer 63275c4186764355a53e17d49746ffe9. Status: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 02:25:05.360735 28252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7343c0d9e76b42198b991afcbeca6199. Latest consensus state: current_term: 2 leader_uuid: "7343c0d9e76b42198b991afcbeca6199" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:05.361080 28252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:05.362883 28251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7343c0d9e76b42198b991afcbeca6199" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:05.363234 28251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:05.363574 28252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7343c0d9e76b42198b991afcbeca6199" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "63275c4186764355a53e17d49746ffe9" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 46797 } } peers { permanent_uuid: "7343c0d9e76b42198b991afcbeca6199" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46471 } attrs { promote: false } } peers { permanent_uuid: "7344bd2a30da48979061abd3c6d4d7cf" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 34721 } attrs { promote: false } } }
I20260808 02:25:05.363868 28252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7344bd2a30da48979061abd3c6d4d7cf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:25:05.381309 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.372499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:36681 (local address 127.0.0.1:59638)
0808 02:25:05.372646 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.372665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.372766 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.374130 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.374135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.374160 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.374359 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.374369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.376304 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.376313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.378937 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.378950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.380534 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.380540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.380760 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.380799 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.380863 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.380999 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 02:25:05.382781 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.372643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.19.231.2:36681)
0808 02:25:05.373797 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.373803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.373821 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:25:05.373889 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.373894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.373970 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.374494 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.374729 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.376085 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.379144 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.380351 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.382141 (+ 1790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.382257 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.382304 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.382445 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":931,"threads_started":1}
I20260808 02:25:05.383891 27961 tablet_service.cc:1467] Processing CreateTablet for tablet 44681e2017cf4f36966366386b5bdd4c (DEFAULT_TABLE table=test-table [id=8f46d086f088403dbd0477f094694c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:05.384516 27961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44681e2017cf4f36966366386b5bdd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:05.396188 28274 tablet_bootstrap.cc:492] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: Bootstrap starting.
I20260808 02:25:05.398996 28274 tablet_bootstrap.cc:654] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:05.404760 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.373298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42421 (local address 127.0.0.1:57124)
0808 02:25:05.373743 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.373764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.373847 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.391668 (+ 17821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.391677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.391723 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.392231 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.392249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.396540 (+ 4291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.396549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.399573 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.399597 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.403548 (+ 3951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.403558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.403879 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.403939 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.404024 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.404232 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260808 02:25:05.406414 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.374499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.19.231.1:42421)
0808 02:25:05.391259 (+ 16760us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.391265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.391305 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:25:05.391380 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.391385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.391462 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.391617 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.393109 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.394611 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.399883 (+ 5272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.401584 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.405523 (+ 3939us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.405709 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.405780 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.405985 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16618,"thread_start_us":16505,"threads_started":1}
I20260808 02:25:05.407449 28274 tablet_bootstrap.cc:492] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: No bootstrap required, opened a new log
I20260808 02:25:05.407691 28274 ts_tablet_manager.cc:1397] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260808 02:25:05.407692 27818 tablet_service.cc:1467] Processing CreateTablet for tablet 44681e2017cf4f36966366386b5bdd4c (DEFAULT_TABLE table=test-table [id=8f46d086f088403dbd0477f094694c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:05.408288 27818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44681e2017cf4f36966366386b5bdd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:05.408670 28274 raft_consensus.cc:348] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.408928 28274 raft_consensus.cc:374] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:05.409030 28274 raft_consensus.cc:729] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1f9169f4c84130b2fa24fb7d9dcf88, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.409337 28274 consensus_queue.cc:260] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.410275 28274 ts_tablet_manager.cc:1428] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:05.413539 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.411006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.2:37601)
0808 02:25:05.413090 (+ 2084us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.413261 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1955,"thread_start_us":1389,"threads_started":1}
I20260808 02:25:05.415941 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.374166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:36813 (local address 127.0.0.1:57784)
0808 02:25:05.381541 (+ 7375us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.381571 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.381660 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.409361 (+ 27701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.409368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.409400 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.409610 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.409622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.411298 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.411304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.413195 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.413212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.414867 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.414873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.415145 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.415207 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.415291 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.415497 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7284,"thread_start_us":28968,"threads_started":1}
I20260808 02:25:05.415980 28029 heartbeater.cc:503] Master 127.19.231.61:46471 was elected leader, sending a full tablet report...
I20260808 02:25:05.416729 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.415516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.2:33489)
0808 02:25:05.416468 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.416587 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":844}
I20260808 02:25:05.417371 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.374293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.19.231.3:36813)
0808 02:25:05.408997 (+ 34704us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.409007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.409031 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:25:05.409109 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.409115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.409204 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.409320 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.409791 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.411104 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.413407 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.414663 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.416450 (+ 1787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.416634 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.416703 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.416973 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34553,"thread_start_us":34432,"threads_started":1}
I20260808 02:25:05.417425 28278 tablet_bootstrap.cc:492] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Bootstrap starting.
I20260808 02:25:05.418283 28108 tablet_service.cc:1467] Processing CreateTablet for tablet 44681e2017cf4f36966366386b5bdd4c (DEFAULT_TABLE table=test-table [id=8f46d086f088403dbd0477f094694c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:05.418780 28108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44681e2017cf4f36966366386b5bdd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:05.419685 28278 tablet_bootstrap.cc:654] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:05.421806 28278 tablet_bootstrap.cc:492] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: No bootstrap required, opened a new log
I20260808 02:25:05.421957 28278 ts_tablet_manager.cc:1397] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 02:25:05.422814 28278 raft_consensus.cc:348] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.423081 28278 raft_consensus.cc:374] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:05.423171 28278 raft_consensus.cc:729] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f14ffc1fc14e6ea70b96582bc99e71, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.423429 28278 consensus_queue.cc:260] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.424348 28278 ts_tablet_manager.cc:1428] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 02:25:05.425700 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.424553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.1:52201)
0808 02:25:05.425162 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.425366 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":319,"threads_started":1}
I20260808 02:25:05.426524 28282 tablet_bootstrap.cc:492] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: Bootstrap starting.
I20260808 02:25:05.427502 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.427061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.1:59417)
0808 02:25:05.427265 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.427348 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 02:25:05.428102 27886 heartbeater.cc:503] Master 127.19.231.61:46471 was elected leader, sending a full tablet report...
I20260808 02:25:05.428625 28282 tablet_bootstrap.cc:654] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:05.430774 28282 tablet_bootstrap.cc:492] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: No bootstrap required, opened a new log
I20260808 02:25:05.430889 28282 ts_tablet_manager.cc:1397] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260808 02:25:05.431473 28282 raft_consensus.cc:348] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.431648 28282 raft_consensus.cc:374] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:05.431721 28282 raft_consensus.cc:729] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b93a6b19e943eea6354fc7971e9e78, State: Initialized, Role: FOLLOWER
I20260808 02:25:05.431958 28282 consensus_queue.cc:260] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.432668 28282 ts_tablet_manager.cc:1428] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:05.435247 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.432954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.3:36573)
0808 02:25:05.433576 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.433785 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":345,"threads_started":1}
I20260808 02:25:05.436245 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.435844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.3:52635)
0808 02:25:05.436011 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.436100 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:25:05.523748 27887 tablet.cc:2369] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:05.628610 28280 raft_consensus.cc:482] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:05.628865 28280 raft_consensus.cc:504] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.630654 28280 leader_election.cc:288] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681), a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813)
I20260808 02:25:05.637238 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.630894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:36681 (local address 127.19.231.1:59289)
0808 02:25:05.631190 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.631216 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.631388 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.631992 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.631998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.632031 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.632412 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.632423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.633679 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.633685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.635217 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.635230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.636564 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.636579 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.636717 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.636743 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.636852 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.636958 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1,"spinlock_wait_cycles":768}
I20260808 02:25:05.637307 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.631088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:59289 (local address 127.19.231.2:36681)
0808 02:25:05.631341 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.631348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.631371 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:25:05.631586 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.631594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.631706 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.631803 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.632563 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.633547 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.635368 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.636383 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.636545 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.636629 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.636897 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.637037 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 02:25:05.638140 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44681e2017cf4f36966366386b5bdd4c" candidate_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" is_pre_election: true
I20260808 02:25:05.638417 27981 raft_consensus.cc:2454] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f14ffc1fc14e6ea70b96582bc99e71 in term 0.
I20260808 02:25:05.638490 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.631206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:45093 (local address 127.19.231.3:36813)
0808 02:25:05.631564 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.631570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.632135 (+ 565us) server_negotiation.cc:407] Connection header received
0808 02:25:05.632430 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.632437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.632534 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.632653 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:05.633247 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.634058 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.636169 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.637288 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.637455 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.637531 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.638094 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.638226 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260808 02:25:05.638664 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.631118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:36813 (local address 127.19.231.1:45093)
0808 02:25:05.632016 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.632036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:05.632170 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:05.632679 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:05.632684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:05.632709 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:05.633046 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.633057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.634222 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.634231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.635868 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:05.635882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.637725 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.637734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:05.637912 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.637952 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:25:05.638067 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 02:25:05.638223 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":20,"thread_start_us":596,"threads_started":1}
I20260808 02:25:05.639144 27772 leader_election.cc:302] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f14ffc1fc14e6ea70b96582bc99e71, fa1f9169f4c84130b2fa24fb7d9dcf88; no voters:
I20260808 02:25:05.639715 28280 raft_consensus.cc:2781] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:05.639580 28128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44681e2017cf4f36966366386b5bdd4c" candidate_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b93a6b19e943eea6354fc7971e9e78" is_pre_election: true
I20260808 02:25:05.639906 28280 raft_consensus.cc:482] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:05.640067 28280 raft_consensus.cc:3037] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:05.640115 28128 raft_consensus.cc:2454] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f14ffc1fc14e6ea70b96582bc99e71 in term 0.
I20260808 02:25:05.642864 28280 raft_consensus.cc:504] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.643581 28280 leader_election.cc:288] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [CANDIDATE]: Term 1 election: Requested vote from peers fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681), a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813)
I20260808 02:25:05.644208 28128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44681e2017cf4f36966366386b5bdd4c" candidate_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b93a6b19e943eea6354fc7971e9e78"
I20260808 02:25:05.644162 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44681e2017cf4f36966366386b5bdd4c" candidate_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88"
I20260808 02:25:05.644431 28128 raft_consensus.cc:3037] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:05.644452 27981 raft_consensus.cc:3037] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:05.648036 27981 raft_consensus.cc:2454] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f14ffc1fc14e6ea70b96582bc99e71 in term 1.
I20260808 02:25:05.648036 28128 raft_consensus.cc:2454] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f14ffc1fc14e6ea70b96582bc99e71 in term 1.
I20260808 02:25:05.648543 27772 leader_election.cc:302] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f14ffc1fc14e6ea70b96582bc99e71, fa1f9169f4c84130b2fa24fb7d9dcf88; no voters:
I20260808 02:25:05.648954 28280 raft_consensus.cc:2781] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:05.649484 28280 raft_consensus.cc:686] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 LEADER]: Becoming Leader. State: Replica: 92f14ffc1fc14e6ea70b96582bc99e71, State: Running, Role: LEADER
I20260808 02:25:05.649988 28280 consensus_queue.cc:237] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } }
I20260808 02:25:05.650545 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.650099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.1:45289)
0808 02:25:05.650285 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.650397 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:25:05.654225 27497 catalog_manager.cc:5495] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1). New cstate: current_term: 1 leader_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92f14ffc1fc14e6ea70b96582bc99e71" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 } health_report { overall_health: UNKNOWN } } }
I20260808 02:25:05.703945 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.698084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.19.231.1:42421)
0808 02:25:05.698304 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.698309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.698326 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:25:05.698418 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.698421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.698498 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.698679 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:05.699223 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.700076 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.701451 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.701970 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.702073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.702530 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:05.703240 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:05.703370 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.703555 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.703626 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:25:05.708330 27981 raft_consensus.cc:1261] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Refusing update from remote peer 92f14ffc1fc14e6ea70b96582bc99e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:05.708501 28128 raft_consensus.cc:1261] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 1 FOLLOWER]: Refusing update from remote peer 92f14ffc1fc14e6ea70b96582bc99e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:05.709205 28288 consensus_queue.cc:1035] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1f9169f4c84130b2fa24fb7d9dcf88" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:05.709632 28280 consensus_queue.cc:1035] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b93a6b19e943eea6354fc7971e9e78" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:05.711851 28176 heartbeater.cc:503] Master 127.19.231.61:46471 was elected leader, sending a full tablet report...
I20260808 02:25:05.712984 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.712306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.2:34009)
0808 02:25:05.712579 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.712727 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260808 02:25:05.713608 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.712350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.3:36593)
0808 02:25:05.713262 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.713391 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":783}
I20260808 02:25:05.723407 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.722914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.3:59951)
0808 02:25:05.723114 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.723214 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 02:25:05.723670 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.722954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.1:45487)
0808 02:25:05.723319 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.723453 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260808 02:25:05.723789 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.723290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.19.231.2:54123)
0808 02:25:05.723537 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.723643 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 02:25:05.740681 27497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:05.740959 27497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:05.748272 27497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7343c0d9e76b42198b991afcbeca6199: Sending DeleteTablet for 3 replicas of tablet 44681e2017cf4f36966366386b5bdd4c
I20260808 02:25:05.749325 27961 tablet_service.cc:1514] Processing DeleteTablet for tablet 44681e2017cf4f36966366386b5bdd4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:05 UTC) from {username='oryx', principal='oryx/127.19.231.61@KRBTEST.COM'} at 127.0.0.1:59638
I20260808 02:25:05.749446 27818 tablet_service.cc:1514] Processing DeleteTablet for tablet 44681e2017cf4f36966366386b5bdd4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:05 UTC) from {username='oryx', principal='oryx/127.19.231.61@KRBTEST.COM'} at 127.0.0.1:57124
I20260808 02:25:05.749559 28108 tablet_service.cc:1514] Processing DeleteTablet for tablet 44681e2017cf4f36966366386b5bdd4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:05 UTC) from {username='oryx', principal='oryx/127.19.231.61@KRBTEST.COM'} at 127.0.0.1:57784
I20260808 02:25:05.755944 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 27669
I20260808 02:25:05.759886 28303 tablet_replica.cc:330] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: stopping tablet replica
I20260808 02:25:05.760363 28303 raft_consensus.cc:2229] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:05.760841 28303 raft_consensus.cc:2258] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:05.762288 28301 tablet_replica.cc:330] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: stopping tablet replica
I20260808 02:25:05.763353 28303 ts_tablet_manager.cc:1905] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.770922 28301 raft_consensus.cc:2229] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:05.771548 28301 raft_consensus.cc:2258] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:05.772506 28302 tablet_replica.cc:330] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: stopping tablet replica
I20260808 02:25:05.772867 28302 raft_consensus.cc:2229] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:05.774022 28301 ts_tablet_manager.cc:1905] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.777148 28302 raft_consensus.cc:2258] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:05.778435 28302 ts_tablet_manager.cc:1905] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:05.782320 28301 ts_tablet_manager.cc:1918] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:05.784606 28303 ts_tablet_manager.cc:1918] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:05.784814 28303 log.cc:1196] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/44681e2017cf4f36966366386b5bdd4c
I20260808 02:25:05.784977 28301 log.cc:1196] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/44681e2017cf4f36966366386b5bdd4c
I20260808 02:25:05.785604 28303 ts_tablet_manager.cc:1939] T 44681e2017cf4f36966366386b5bdd4c P a5b93a6b19e943eea6354fc7971e9e78: Deleting consensus metadata
I20260808 02:25:05.786198 28301 ts_tablet_manager.cc:1939] T 44681e2017cf4f36966366386b5bdd4c P fa1f9169f4c84130b2fa24fb7d9dcf88: Deleting consensus metadata
I20260808 02:25:05.786665 28302 ts_tablet_manager.cc:1918] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:05.786851 28302 log.cc:1196] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/44681e2017cf4f36966366386b5bdd4c
I20260808 02:25:05.787401 28302 ts_tablet_manager.cc:1939] T 44681e2017cf4f36966366386b5bdd4c P 92f14ffc1fc14e6ea70b96582bc99e71: Deleting consensus metadata
I20260808 02:25:05.787750 27484 catalog_manager.cc:4841] TS a5b93a6b19e943eea6354fc7971e9e78 (127.19.231.3:36813): tablet 44681e2017cf4f36966366386b5bdd4c (table test-table [id=8f46d086f088403dbd0477f094694c35]) successfully deleted
I20260808 02:25:05.788548 27483 catalog_manager.cc:4841] TS 92f14ffc1fc14e6ea70b96582bc99e71 (127.19.231.1:42421): tablet 44681e2017cf4f36966366386b5bdd4c (table test-table [id=8f46d086f088403dbd0477f094694c35]) successfully deleted
I20260808 02:25:05.789371 27483 catalog_manager.cc:4841] TS fa1f9169f4c84130b2fa24fb7d9dcf88 (127.19.231.2:36681): tablet 44681e2017cf4f36966366386b5bdd4c (table test-table [id=8f46d086f088403dbd0477f094694c35]) successfully deleted
I20260808 02:25:05.790668 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.757505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.19.231.1:42421)
0808 02:25:05.757702 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.757709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.769574 (+ 11865us) server_negotiation.cc:407] Connection header received
0808 02:25:05.770081 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.770088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.770199 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.770280 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:05.772424 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.773703 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.775641 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.776397 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.776552 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.780670 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:05.780691 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:05.780711 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:05.780757 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:05.786113 (+ 5356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.787238 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.787252 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.787256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.787457 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.788041 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.788046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.788049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.789030 (+ 981us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:05.789350 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.789999 (+ 649us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.790256 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:25:05.791422 27879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:05.905129 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 27669
I20260808 02:25:05.928571 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 27896
I20260808 02:25:05.942742 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.929340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.19.231.2:36681)
0808 02:25:05.929572 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:05.929577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:05.929596 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:25:05.929752 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:05.929757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:05.929827 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:05.929913 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:05.931223 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.931985 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.933221 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:05.933624 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:05.933780 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:05.936538 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:05.936553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:05.936570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:05.936617 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:05.939665 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.940434 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.940444 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.940447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.940644 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:05.941037 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:05.941042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:05.941046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:05.941444 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:05.941771 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:05.942045 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 02:25:05.942283 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:25:05.943415 28022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:05.944888 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.944014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:53632)
0808 02:25:05.944551 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.944657 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":265,"threads_started":1}
I20260808 02:25:05.946272 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:05.945895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:46797 (local address 127.0.0.1:53636)
0808 02:25:05.946031 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:05.946110 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 02:25:06.038893 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 27896
I20260808 02:25:06.061421 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 28041
I20260808 02:25:06.076784 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:06.062313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.19.231.3:36813)
0808 02:25:06.062509 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:06.062517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:06.062538 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:25:06.062977 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:06.062984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:06.063103 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:06.063230 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:06.064880 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:06.065713 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:06.067327 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:06.067853 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:06.068003 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:06.070855 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:06.070871 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:06.070888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:06.070922 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:06.073740 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:06.074464 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:06.074472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:06.074474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:06.074751 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:06.075144 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:06.075149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:06.075152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:06.075555 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:06.075873 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:06.076175 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:25:06.076422 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:25:06.077479 28169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:06.199779 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 28041
I20260808 02:25:06.228228 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 27355
I20260808 02:25:06.229430 27527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:06.353848 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 27355
W20260808 02:25:06.379240 27682 connection.cc:535] server connection from 127.0.0.1:36580 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:25:06.379657 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 27461
I20260808 02:25:06.380789 27730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:06.499954 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 27461
2026-08-08T02:25:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10705 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:06 dist-test-slave-pgkr krb5kdc[28314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:06 dist-test-slave-pgkr krb5kdc[28314](info): set up 2 sockets
Aug 08 02:25:06 dist-test-slave-pgkr krb5kdc[28314](info): commencing operation
krb5kdc: starting...
W20260808 02:25:07.657367 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:25:07 dist-test-slave-pgkr krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab.
Entry for principal oryx/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab.
I20260808 02:25:07.933763 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:39901
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab
--principal=oryx/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:39901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:08.278431 28328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:08.278756 28328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:08.278848 28328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:08.286114 28328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:08.286226 28328 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 02:25:08.286275 28328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:08.286361 28328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:08.286413 28328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:08.286497 28328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:08.296509 28328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:39901
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.62.keytab
--principal=oryx/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:39901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:08.297837 28328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:08.299177 28328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:08.311271 28333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:08.311234 28334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:08.311244 28336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:08.312014 28328 server_base.cc:1034] running on GCE node
I20260808 02:25:08.312830 28328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:08.314232 28328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:08.315469 28328 hybrid_clock.cc:648] HybridClock initialized: now 1786155908315406 us; error 39 us; skew 500 ppm
I20260808 02:25:08.315802 28328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:08 dist-test-slave-pgkr krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:08.323665 28328 init.cc:377] Logged in from keytab as oryx/127.19.231.62@KRBTEST.COM (short username oryx)
I20260808 02:25:08.325197 28328 webserver.cc:460] Webserver started at http://127.19.231.62:45599/ using document root <none> and password file <none>
I20260808 02:25:08.325907 28328 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:08.326037 28328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:08.326364 28328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:08.329341 28328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "573a0de1e1cd4cfbb22266e7642a621f"
format_stamp: "Formatted at 2026-08-08 02:25:08 on dist-test-slave-pgkr"
server_key: "1d4d5231074c6112c977df97ae31626c"
server_key_iv: "0549c27d7641a9894e96dd4a1634f635"
server_key_version: "encryptionkey@0"
I20260808 02:25:08.330216 28328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "573a0de1e1cd4cfbb22266e7642a621f"
format_stamp: "Formatted at 2026-08-08 02:25:08 on dist-test-slave-pgkr"
server_key: "1d4d5231074c6112c977df97ae31626c"
server_key_iv: "0549c27d7641a9894e96dd4a1634f635"
server_key_version: "encryptionkey@0"
I20260808 02:25:08.335477 28328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 02:25:08.339027 28343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:08.340478 28328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 02:25:08.340680 28328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "573a0de1e1cd4cfbb22266e7642a621f"
format_stamp: "Formatted at 2026-08-08 02:25:08 on dist-test-slave-pgkr"
server_key: "1d4d5231074c6112c977df97ae31626c"
server_key_iv: "0549c27d7641a9894e96dd4a1634f635"
server_key_version: "encryptionkey@0"
I20260808 02:25:08.340902 28328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:08.359335 28328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:08.363500 28328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:08.363798 28328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:08.386014 28328 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:39901
I20260808 02:25:08.386022 28405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:39901 every 8 connection(s)
I20260808 02:25:08.387429 28328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:08.391228 28406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:08.391666 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 28328
I20260808 02:25:08.391888 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:25:08.392791 20380 external_mini_cluster.cc:1428] Setting key 3767781b2d664b38e35df5bd841b4846
I20260808 02:25:08.405530 28406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Bootstrap starting.
Aug 08 02:25:08 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:08.409242 28406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:08.410323 28406 log.cc:824] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:08.413169 28406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: No bootstrap required, opened a new log
I20260808 02:25:08.416585 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:08.394972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.19.231.62:39901)
0808 02:25:08.395808 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:08.395819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:08.395865 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:25:08.397702 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:08.397725 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:08.398113 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:08.398456 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:08.400329 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:08.401594 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:08.403455 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:08.404089 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:08.404228 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:08.408904 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:08.408912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:08.408926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:08.408959 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:08.412065 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:08.412794 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:08.412801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:08.412806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:08.412979 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:08.413404 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:08.413409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:08.413414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:08.414130 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:08.414427 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:08.414791 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 02:25:08.415237 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":393,"threads_started":1}
I20260808 02:25:08.421718 28406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573a0de1e1cd4cfbb22266e7642a621f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39901 } }
I20260808 02:25:08.422037 28406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:08.422122 28406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573a0de1e1cd4cfbb22266e7642a621f, State: Initialized, Role: FOLLOWER
I20260808 02:25:08.422590 28406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573a0de1e1cd4cfbb22266e7642a621f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39901 } }
I20260808 02:25:08.422936 28406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:08.423118 28406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:08.423316 28406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:08.425498 28406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573a0de1e1cd4cfbb22266e7642a621f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39901 } }
I20260808 02:25:08.426015 28406 leader_election.cc:302] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 573a0de1e1cd4cfbb22266e7642a621f; no voters:
I20260808 02:25:08.426672 28406 leader_election.cc:288] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:08.426802 28411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:08.427543 28411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [term 1 LEADER]: Becoming Leader. State: Replica: 573a0de1e1cd4cfbb22266e7642a621f, State: Running, Role: LEADER
I20260808 02:25:08.428133 28411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573a0de1e1cd4cfbb22266e7642a621f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39901 } }
I20260808 02:25:08.428864 28406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:08.431391 28412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "573a0de1e1cd4cfbb22266e7642a621f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573a0de1e1cd4cfbb22266e7642a621f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39901 } } }
I20260808 02:25:08.431468 28413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573a0de1e1cd4cfbb22266e7642a621f. Latest consensus state: current_term: 1 leader_uuid: "573a0de1e1cd4cfbb22266e7642a621f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573a0de1e1cd4cfbb22266e7642a621f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39901 } } }
I20260808 02:25:08.431895 28412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:08.431974 28413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:08.433391 28421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:08.438974 28421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:08.447958 28421 catalog_manager.cc:1347] Generated new cluster ID: 5bf7f46a692a4d8fbf66bac90e721632
I20260808 02:25:08.448071 28421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:08.470268 28421 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:08.472138 28421 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:08.485360 28421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Generated new TSK 0
I20260808 02:25:08.486198 28421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab.
Entry for principal oryx/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab.
I20260808 02:25:08.558382 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39901
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab
--principal=oryx/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:08.909345 28433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:08.909650 28433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:08.909817 28433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:08.917308 28433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:08.917457 28433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:08.917599 28433 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:25:08.917732 28433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:25:08.927498 28433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab
--principal=oryx/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:39901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:08.928812 28433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:08.930271 28433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:08.944132 28439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:08.944191 28440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:08.944217 28442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:08.945176 28433 server_base.cc:1034] running on GCE node
I20260808 02:25:08.945919 28433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:08.946887 28433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:08.948149 28433 hybrid_clock.cc:648] HybridClock initialized: now 1786155908948080 us; error 58 us; skew 500 ppm
I20260808 02:25:08.948477 28433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:08 dist-test-slave-pgkr krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:08.951817 28433 init.cc:377] Logged in from keytab as oryx/127.19.231.1@KRBTEST.COM (short username oryx)
I20260808 02:25:08.953384 28433 webserver.cc:460] Webserver started at http://127.19.231.1:43205/ using document root <none> and password file <none>
I20260808 02:25:08.954198 28433 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:08.954322 28433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:08.954731 28433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:08.957556 28433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f"
format_stamp: "Formatted at 2026-08-08 02:25:08 on dist-test-slave-pgkr"
server_key: "46c8aa16509156f6b5475d5b1eaceaae"
server_key_iv: "825f4053b4cef7b4cf99d27e1ec7c5cb"
server_key_version: "encryptionkey@0"
I20260808 02:25:08.958348 28433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f"
format_stamp: "Formatted at 2026-08-08 02:25:08 on dist-test-slave-pgkr"
server_key: "46c8aa16509156f6b5475d5b1eaceaae"
server_key_iv: "825f4053b4cef7b4cf99d27e1ec7c5cb"
server_key_version: "encryptionkey@0"
I20260808 02:25:08.963318 28433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260808 02:25:08.966851 28449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:08.968181 28433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:25:08.968375 28433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f"
format_stamp: "Formatted at 2026-08-08 02:25:08 on dist-test-slave-pgkr"
server_key: "46c8aa16509156f6b5475d5b1eaceaae"
server_key_iv: "825f4053b4cef7b4cf99d27e1ec7c5cb"
server_key_version: "encryptionkey@0"
I20260808 02:25:08.968662 28433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:08.987277 28433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:08.991246 28433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:08.991523 28433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:08.994168 28433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:08.994285 28433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:08.994457 28433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:08.994535 28433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:25:09 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:09.020870 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:08.994969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:52703 (local address 127.19.231.62:39901)
0808 02:25:08.995544 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:08.995551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:08.996739 (+ 1188us) server_negotiation.cc:407] Connection header received
0808 02:25:08.999903 (+ 3164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:08.999910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:09.000012 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:09.000147 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:09.002454 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.003789 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.007057 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.007613 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.007781 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.012981 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:09.012986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:09.012989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:09.013022 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:09.016918 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.018021 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.018027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.018031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.018173 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.018665 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.018670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.018674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.018946 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:09.019148 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:09.020241 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0808 02:25:09.020472 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":338,"threads_started":1}
I20260808 02:25:09.021945 28459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:08.995632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39901 (local address 127.19.231.1:52703)
0808 02:25:08.996544 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:08.996611 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:08.999481 (+ 2870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:09.000511 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:09.000527 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:09.001362 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:09.002224 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.002260 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.004026 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.004034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.006880 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.006899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.007909 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.007924 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.008314 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.009207 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:09.009245 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:09.012744 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:09.017080 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.017091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.017112 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.017836 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.018296 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.018301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.018304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.018513 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.019237 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:09.019251 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:09.019751 (+ 500us) client_negotiation.cc:769] Sending connection context
0808 02:25:09.020139 (+ 388us) client_negotiation.cc:240] Negotiation successful
0808 02:25:09.020692 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":369,"threads_started":1}
I20260808 02:25:09.045656 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.028487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:44101 (local address 127.19.231.62:39901)
0808 02:25:09.028649 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:09.028653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:09.029814 (+ 1161us) server_negotiation.cc:407] Connection header received
0808 02:25:09.030048 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:09.030052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:09.030116 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:09.030196 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:09.031742 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.032443 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.035066 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.035657 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.035794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.038557 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:09.038562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:09.038564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:09.038593 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:09.041674 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.042759 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.042767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.042773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.042948 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.043452 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.043457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.043460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.043879 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:09.044042 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:09.045057 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0808 02:25:09.045324 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:25:09.045765 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.028677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39901 (local address 127.19.231.1:44101)
0808 02:25:09.029686 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:09.029712 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:09.029888 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:09.030285 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:09.030292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:09.031081 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:09.031560 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.031576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.032572 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.032581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.034891 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.034918 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.035906 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.035916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.036180 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.036902 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:09.036934 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:09.038363 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:09.041819 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.041825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.041829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.042547 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.043098 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.043105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.043110 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.043311 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.044107 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:09.044114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:09.044374 (+ 260us) client_negotiation.cc:769] Sending connection context
0808 02:25:09.044561 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 02:25:09.045034 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":580,"threads_started":1}
I20260808 02:25:09.053817 28433 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:46063
I20260808 02:25:09.053839 28575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:46063 every 8 connection(s)
I20260808 02:25:09.055266 28433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:09.057330 28576 heartbeater.cc:347] Connected to a master server at 127.19.231.62:39901
I20260808 02:25:09.057565 28576 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:09.057770 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 28433
I20260808 02:25:09.057979 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:25:09.058336 28576 heartbeater.cc:511] Master 127.19.231.62:39901 requested a full tablet report, sending...
I20260808 02:25:09.058732 20380 external_mini_cluster.cc:1428] Setting key 6ce2803c7abb7cdc9f6d77713486c084
I20260808 02:25:09.060405 28361 ts_manager.cc:194] Registered new tserver with Master: 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063)
I20260808 02:25:09.061841 28361 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:52703
WARNING: no policy specified for oryx/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab.
Entry for principal oryx/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab.
I20260808 02:25:09.118013 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39901
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab
--principal=oryx/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:09.460347 28580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:09.460633 28580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:09.460799 28580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:09.468740 28580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:09.468847 28580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:09.468992 28580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:25:09.469103 28580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:25:09.479264 28580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab
--principal=oryx/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:39901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:09.480597 28580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:09.481968 28580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:09.495698 28585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:09.495831 28586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:09.495851 28588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:09.496624 28580 server_base.cc:1034] running on GCE node
I20260808 02:25:09.497241 28580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:09.498189 28580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:09.499459 28580 hybrid_clock.cc:648] HybridClock initialized: now 1786155909499384 us; error 43 us; skew 500 ppm
I20260808 02:25:09.499812 28580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:09 dist-test-slave-pgkr krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155909, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:09.503219 28580 init.cc:377] Logged in from keytab as oryx/127.19.231.2@KRBTEST.COM (short username oryx)
I20260808 02:25:09.504703 28580 webserver.cc:460] Webserver started at http://127.19.231.2:33271/ using document root <none> and password file <none>
I20260808 02:25:09.505414 28580 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:09.505548 28580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:09.505892 28580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:09.508984 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "c955d9ea567d48b4b04be23fa4800296"
format_stamp: "Formatted at 2026-08-08 02:25:09 on dist-test-slave-pgkr"
server_key: "3897c3a54789a8ec863da6b4126a8309"
server_key_iv: "fdc426d346a67c36e806fd0ca6a01d7c"
server_key_version: "encryptionkey@0"
I20260808 02:25:09.509805 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "c955d9ea567d48b4b04be23fa4800296"
format_stamp: "Formatted at 2026-08-08 02:25:09 on dist-test-slave-pgkr"
server_key: "3897c3a54789a8ec863da6b4126a8309"
server_key_iv: "fdc426d346a67c36e806fd0ca6a01d7c"
server_key_version: "encryptionkey@0"
I20260808 02:25:09.515235 28580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260808 02:25:09.518882 28595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:09.520324 28580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:25:09.520574 28580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "c955d9ea567d48b4b04be23fa4800296"
format_stamp: "Formatted at 2026-08-08 02:25:09 on dist-test-slave-pgkr"
server_key: "3897c3a54789a8ec863da6b4126a8309"
server_key_iv: "fdc426d346a67c36e806fd0ca6a01d7c"
server_key_version: "encryptionkey@0"
I20260808 02:25:09.520833 28580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:09.548671 28580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:09.553517 28580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:09.553825 28580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:09.556694 28580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:09.556810 28580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:09.556981 28580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:09.557080 28580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:25:09 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155909, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:09.583937 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.557375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:58339 (local address 127.19.231.62:39901)
0808 02:25:09.558058 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:09.558065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:09.559249 (+ 1184us) server_negotiation.cc:407] Connection header received
0808 02:25:09.561521 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:09.561527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:09.561628 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:09.561760 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:09.564601 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.565616 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.568974 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.569539 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.569693 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.575548 (+ 5855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:09.575558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:09.575565 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:09.575613 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:09.579716 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.580775 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.580780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.580782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.580911 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.581404 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.581415 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.581418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.581731 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:09.581915 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:09.583141 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0808 02:25:09.583440 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":436,"threads_started":1}
I20260808 02:25:09.585887 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.558130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39901 (local address 127.19.231.2:58339)
0808 02:25:09.559073 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:09.559124 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:09.561217 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:09.562118 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:09.562132 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:09.563299 (+ 1167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:09.564375 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.564408 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.565801 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.565808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.568790 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.568808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.569826 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.569838 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.570224 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.571159 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:09.571200 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:09.575248 (+ 4048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:09.579910 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.579921 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.579943 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.580593 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.581033 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.581038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.581042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.581267 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.581893 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:09.581909 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:09.582538 (+ 629us) client_negotiation.cc:769] Sending connection context
0808 02:25:09.583005 (+ 467us) client_negotiation.cc:240] Negotiation successful
0808 02:25:09.583774 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":433,"threads_started":1}
I20260808 02:25:09.605216 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.591834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39901 (local address 127.19.231.2:54921)
0808 02:25:09.592443 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:09.592460 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:09.592612 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:09.592925 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:09.592930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:09.593390 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:09.593671 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.593679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.594655 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.594663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.596445 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:09.596458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.597228 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.597234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:09.597400 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.597999 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:09.598016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:09.598965 (+ 949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:09.602372 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.602377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.602381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.602990 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.603461 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:09.603464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:09.603466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:09.603582 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:09.604403 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:09.604406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:09.604557 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 02:25:09.604679 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 02:25:09.604899 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":351,"threads_started":1}
I20260808 02:25:09.605666 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:09.591731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:54921 (local address 127.19.231.62:39901)
0808 02:25:09.591884 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:09.591889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:09.592543 (+ 654us) server_negotiation.cc:407] Connection header received
0808 02:25:09.592738 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:09.592742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:09.592800 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:09.592902 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:09.593808 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.594497 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.596603 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:09.597031 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:09.597167 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:09.599174 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:09.599179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:09.599182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:09.599209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:09.602236 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.603131 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.603139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.603146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.603330 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:09.603705 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:09.603712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:09.603719 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:09.604278 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:09.604469 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:09.604862 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 02:25:09.605181 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:25:09.612967 28580 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:38911
I20260808 02:25:09.613008 28719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:38911 every 8 connection(s)
I20260808 02:25:09.614223 28580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:09.616389 28720 heartbeater.cc:347] Connected to a master server at 127.19.231.62:39901
I20260808 02:25:09.616577 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 28580
I20260808 02:25:09.616710 28720 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:09.616758 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:25:09.617228 20380 external_mini_cluster.cc:1428] Setting key 12bde98f6da382c6ac178c9e3840a923
I20260808 02:25:09.617410 28720 heartbeater.cc:511] Master 127.19.231.62:39901 requested a full tablet report, sending...
I20260808 02:25:09.619253 28360 ts_manager.cc:194] Registered new tserver with Master: c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911)
I20260808 02:25:09.620231 28360 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:58339
WARNING: no policy specified for oryx/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab.
Entry for principal oryx/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab.
I20260808 02:25:09.677467 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39901
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab
--principal=oryx/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:10.020352 28724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:10.020702 28724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:10.020885 28724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:10.028627 28724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:10.028800 28724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:10.028947 28724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 02:25:10.029098 28724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:25:10.039023 28724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab
--principal=oryx/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.231.62:39901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:10.040468 28724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:10.041869 28724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:10.055747 28733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:10.055873 28731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:10.055719 28730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:10.057323 28724 server_base.cc:1034] running on GCE node
I20260808 02:25:10.058051 28724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:10.059302 28724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:10.060578 28724 hybrid_clock.cc:648] HybridClock initialized: now 1786155910060466 us; error 89 us; skew 500 ppm
I20260808 02:25:10.060890 28724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:10 dist-test-slave-pgkr krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:10.064286 28724 init.cc:377] Logged in from keytab as oryx/127.19.231.3@KRBTEST.COM (short username oryx)
I20260808 02:25:10.064574 28576 heartbeater.cc:503] Master 127.19.231.62:39901 was elected leader, sending a full tablet report...
I20260808 02:25:10.066001 28724 webserver.cc:460] Webserver started at http://127.19.231.3:34887/ using document root <none> and password file <none>
I20260808 02:25:10.066947 28724 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:10.067067 28724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:10.067430 28724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:10.070497 28724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "7e68e71d9d38456a9bf6d4e4887db304"
format_stamp: "Formatted at 2026-08-08 02:25:10 on dist-test-slave-pgkr"
server_key: "f61147bd63c1da832fe952707d166196"
server_key_iv: "9d3732fc6bda4e9e1d443c0eb270c50a"
server_key_version: "encryptionkey@0"
I20260808 02:25:10.071368 28724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "7e68e71d9d38456a9bf6d4e4887db304"
format_stamp: "Formatted at 2026-08-08 02:25:10 on dist-test-slave-pgkr"
server_key: "f61147bd63c1da832fe952707d166196"
server_key_iv: "9d3732fc6bda4e9e1d443c0eb270c50a"
server_key_version: "encryptionkey@0"
I20260808 02:25:10.076864 28724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 02:25:10.080384 28740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:10.081745 28724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:25:10.081925 28724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "7e68e71d9d38456a9bf6d4e4887db304"
format_stamp: "Formatted at 2026-08-08 02:25:10 on dist-test-slave-pgkr"
server_key: "f61147bd63c1da832fe952707d166196"
server_key_iv: "9d3732fc6bda4e9e1d443c0eb270c50a"
server_key_version: "encryptionkey@0"
I20260808 02:25:10.082162 28724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:10.096946 28724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:10.101377 28724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:10.101676 28724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:10.104549 28724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:10.104679 28724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:10.104861 28724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:10.104949 28724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:25:10 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:10.130679 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:10.105084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:44025 (local address 127.19.231.62:39901)
0808 02:25:10.105300 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:10.105307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:10.106827 (+ 1520us) server_negotiation.cc:407] Connection header received
0808 02:25:10.109288 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:10.109296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:10.109406 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:10.109541 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:10.111930 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.113046 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.115961 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.116374 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.116472 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:10.122388 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:10.122397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:10.122404 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:10.122455 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:10.126742 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:10.127560 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:10.127568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:10.127574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:10.127783 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:10.128243 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:10.128251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:10.128256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:10.128725 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:10.128963 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:10.129731 (+ 768us) server_negotiation.cc:299] Negotiation successful
0808 02:25:10.130086 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:25:10.131414 28751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:10.105740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39901 (local address 127.19.231.3:44025)
0808 02:25:10.106641 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:10.106686 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:10.108917 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:10.109852 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:10.109867 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:10.110849 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:10.111704 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:10.111742 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.113290 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.113294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:10.115742 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:10.115760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.116652 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.116661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:10.116955 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:10.117933 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:10.117967 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:10.122049 (+ 4082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:10.126934 (+ 4885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:10.126946 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:10.126970 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:10.127430 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:10.127902 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:10.127905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:10.127907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:10.128118 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:10.128890 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:10.128895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:10.129315 (+ 420us) client_negotiation.cc:769] Sending connection context
0808 02:25:10.129552 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 02:25:10.129973 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":394,"threads_started":1}
I20260808 02:25:10.160079 28724 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:41513
I20260808 02:25:10.160125 28862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:41513 every 8 connection(s)
I20260808 02:25:10.161847 28724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:10.161893 28837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:10.142816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39901 (local address 127.19.231.3:36951)
0808 02:25:10.147232 (+ 4416us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:10.147259 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:10.147461 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:10.147826 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:10.147830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:10.148348 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:10.148644 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:10.148653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.149816 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.149829 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:10.151173 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:10.151186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.152087 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.152095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:10.152331 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:10.153240 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:10.153273 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:10.154755 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:10.159129 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:10.159134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:10.159140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:10.159717 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:10.160211 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:10.160216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:10.160219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:10.160398 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:10.161043 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:10.161047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:10.161214 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 02:25:10.161318 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:25:10.161561 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4207,"thread_start_us":4009,"threads_started":1}
I20260808 02:25:10.162086 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:10.142850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:36951 (local address 127.19.231.62:39901)
0808 02:25:10.143097 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:10.143117 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:10.147378 (+ 4261us) server_negotiation.cc:407] Connection header received
0808 02:25:10.147591 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:10.147595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:10.147668 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:10.147789 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:10.148794 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.149660 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.151318 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.151876 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.152011 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:10.154987 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:10.154997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:10.155004 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:10.155049 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:10.158950 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:10.159874 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:10.159882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:10.159888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:10.160077 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:10.160538 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:10.160542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:10.160545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:10.160914 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:10.161087 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:10.161479 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 02:25:10.161723 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:25:10.164379 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 28724
I20260808 02:25:10.164559 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:25:10.165040 20380 external_mini_cluster.cc:1428] Setting key dc3b6d9749ebf0a905c3785a573c4bbc
I20260808 02:25:10.164052 28863 heartbeater.cc:347] Connected to a master server at 127.19.231.62:39901
I20260808 02:25:10.166638 28863 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:10.167423 28863 heartbeater.cc:511] Master 127.19.231.62:39901 requested a full tablet report, sending...
I20260808 02:25:10.168794 28360 ts_manager.cc:194] Registered new tserver with Master: 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513)
I20260808 02:25:10.169574 28360 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:44025
I20260808 02:25:10.180747 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 02:25:10 dist-test-slave-pgkr krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 02:25:10 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:10 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:10.623373 28720 heartbeater.cc:503] Master 127.19.231.62:39901 was elected leader, sending a full tablet report...
I20260808 02:25:11.171962 28863 heartbeater.cc:503] Master 127.19.231.62:39901 was elected leader, sending a full tablet report...
W20260808 02:25:11.445817 28871 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.205s user 0.008s sys 0.012s
W20260808 02:25:11.447161 28604 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:10.234060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.19.231.62:39901)
0808 02:25:10.234285 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:10.234289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:10.234942 (+ 653us) server_negotiation.cc:407] Connection header received
0808 02:25:10.235152 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:10.235178 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:10.235246 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:10.235340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:10.236871 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.237597 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.239011 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:10.239395 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:10.239501 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.446874 (+1207373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60530: BlockingRecv error: recv got EOF from 127.0.0.1:60530 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 02:25:11 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:11.465862 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.451614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.19.231.62:39901)
0808 02:25:11.451819 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:11.451824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:11.452183 (+ 359us) server_negotiation.cc:407] Connection header received
0808 02:25:11.452349 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:11.452353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:11.452418 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:11.452554 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:11.453813 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.454570 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.455864 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.456299 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.456397 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.460656 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:11.460661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:11.460664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:11.460701 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:11.463414 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:11.464067 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:11.464071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:11.464073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:11.464203 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:11.464567 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:11.464572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:11.464576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:11.464886 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:11.465118 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:11.465349 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 02:25:11.465520 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:25:11.470125 28360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:11.477423 28360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:11.503566 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.490282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:41513 (local address 127.0.0.1:35712)
0808 02:25:11.491072 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.491120 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:11.491346 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:11.492954 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:11.492967 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:11.493010 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:11.493540 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.493556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.496121 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.496131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.499413 (+ 3282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.499436 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.502403 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.502413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.502765 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.502823 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 02:25:11.502959 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 02:25:11.503177 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":378,"threads_started":1}
I20260808 02:25:11.504730 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.490842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.19.231.3:41513)
0808 02:25:11.492071 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:11.492084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:11.492107 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:25:11.492186 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:11.492192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:11.492444 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:11.492646 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:11.493798 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.495858 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.499654 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.501716 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.501890 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.502189 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:11.504060 (+ 1871us) server_negotiation.cc:299] Negotiation successful
0808 02:25:11.504332 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":988,"threads_started":1}
I20260808 02:25:11.509814 28792 tablet_service.cc:1467] Processing CreateTablet for tablet ecaed749af29461bab471e208581afa9 (DEFAULT_TABLE table=test-table [id=9eb7f84ebd94486faa1edebf6d738f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:11.511548 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaed749af29461bab471e208581afa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:11.512132 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.490333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46063 (local address 127.0.0.1:43750)
0808 02:25:11.501978 (+ 11645us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.502005 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:11.502909 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:11.504105 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:11.504114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:11.504149 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:11.504617 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.504631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.507660 (+ 3029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.507667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.509384 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.509399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.511352 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.511356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.511466 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.511486 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 02:25:11.511597 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 02:25:11.511783 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11432,"thread_start_us":11856,"threads_started":1}
I20260808 02:25:11.512118 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.490670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.19.231.1:46063)
0808 02:25:11.491870 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:11.491880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:11.502145 (+ 10265us) server_negotiation.cc:407] Connection header received
0808 02:25:11.503167 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:11.503180 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:11.503530 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:11.503763 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:11.504866 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.507458 (+ 2592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.509578 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.510965 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.511050 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.511202 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:11.511643 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 02:25:11.511820 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":862,"threads_started":1}
I20260808 02:25:11.513389 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.492973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.19.231.2:38911)
0808 02:25:11.505138 (+ 12165us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:11.505160 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:11.505804 (+ 644us) server_negotiation.cc:407] Connection header received
0808 02:25:11.505937 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:11.505945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:11.506212 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:11.506427 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:11.506959 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.508449 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.510635 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.512112 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.512249 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.512460 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:11.512943 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 02:25:11.513113 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11979,"thread_start_us":12180,"threads_started":1}
I20260808 02:25:11.514806 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.492634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:38911 (local address 127.0.0.1:41916)
0808 02:25:11.505672 (+ 13038us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.505693 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:11.505801 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:11.506425 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:11.506429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:11.506453 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:11.506819 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.506828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.508618 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.508623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.510414 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.510429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.512563 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.512569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.512748 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.512778 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:25:11.512884 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:25:11.513016 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12915,"mutex_wait_us":1,"thread_start_us":13480,"threads_started":1}
I20260808 02:25:11.516053 28502 tablet_service.cc:1467] Processing CreateTablet for tablet ecaed749af29461bab471e208581afa9 (DEFAULT_TABLE table=test-table [id=9eb7f84ebd94486faa1edebf6d738f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:11.517616 28502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaed749af29461bab471e208581afa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:11.519939 28648 tablet_service.cc:1467] Processing CreateTablet for tablet ecaed749af29461bab471e208581afa9 (DEFAULT_TABLE table=test-table [id=9eb7f84ebd94486faa1edebf6d738f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:11.521373 28648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaed749af29461bab471e208581afa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:11.528716 28890 tablet_bootstrap.cc:492] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Bootstrap starting.
I20260808 02:25:11.532209 28891 tablet_bootstrap.cc:492] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Bootstrap starting.
I20260808 02:25:11.532568 28890 tablet_bootstrap.cc:654] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:11.534045 28890 log.cc:824] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:11.535835 28891 tablet_bootstrap.cc:654] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:11.536824 28891 log.cc:824] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:11.539618 28891 tablet_bootstrap.cc:492] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: No bootstrap required, opened a new log
I20260808 02:25:11.539873 28891 ts_tablet_manager.cc:1397] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 02:25:11.540709 28890 tablet_bootstrap.cc:492] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: No bootstrap required, opened a new log
I20260808 02:25:11.540954 28890 ts_tablet_manager.cc:1397] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260808 02:25:11.542807 28892 tablet_bootstrap.cc:492] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Bootstrap starting.
I20260808 02:25:11.546348 28892 tablet_bootstrap.cc:654] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:11.547664 28892 log.cc:824] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:11.549628 28890 raft_consensus.cc:348] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.549625 28891 raft_consensus.cc:348] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.549989 28890 raft_consensus.cc:374] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:11.550081 28891 raft_consensus.cc:374] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:11.550102 28890 raft_consensus.cc:729] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e68e71d9d38456a9bf6d4e4887db304, State: Initialized, Role: FOLLOWER
I20260808 02:25:11.550200 28891 raft_consensus.cc:729] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179ff45c551e4d5b8d4dd88f09ab4e3f, State: Initialized, Role: FOLLOWER
I20260808 02:25:11.550796 28890 consensus_queue.cc:260] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.550957 28892 tablet_bootstrap.cc:492] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: No bootstrap required, opened a new log
I20260808 02:25:11.550956 28891 consensus_queue.cc:260] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.551251 28892 ts_tablet_manager.cc:1397] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260808 02:25:11.553630 28890 ts_tablet_manager.cc:1428] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260808 02:25:11.556432 28891 ts_tablet_manager.cc:1428] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20260808 02:25:11.559058 28577 tablet.cc:2369] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:11.561707 28892 raft_consensus.cc:348] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.562085 28892 raft_consensus.cc:374] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:11.562193 28892 raft_consensus.cc:729] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c955d9ea567d48b4b04be23fa4800296, State: Initialized, Role: FOLLOWER
I20260808 02:25:11.562846 28892 consensus_queue.cc:260] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.564174 28892 ts_tablet_manager.cc:1428] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
W20260808 02:25:11.617291 28721 tablet.cc:2369] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:25:11.667510 28864 tablet.cc:2369] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:11.715839 28896 raft_consensus.cc:482] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:11.716231 28896 raft_consensus.cc:504] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.718032 28896 leader_election.cc:288] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911)
I20260808 02:25:11.725319 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.718651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:47129 (local address 127.19.231.2:38911)
0808 02:25:11.718916 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:11.718922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:11.719656 (+ 734us) server_negotiation.cc:407] Connection header received
0808 02:25:11.719876 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:11.719880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:11.719937 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:11.720038 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:11.720631 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.721356 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.723422 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.724323 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.724500 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.724571 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:11.724902 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 02:25:11.725033 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:25:11.725598 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.718508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:38911 (local address 127.19.231.3:47129)
0808 02:25:11.719498 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.719512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:11.719725 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:11.720052 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:11.720057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:11.720079 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:11.720455 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.720467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.721486 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.721493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.723257 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.723270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.724516 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.724521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.724676 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.724700 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 02:25:11.724894 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 02:25:11.725138 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":103,"thread_start_us":723,"threads_started":1}
I20260808 02:25:11.725838 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.718373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:58529 (local address 127.19.231.1:46063)
0808 02:25:11.718663 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:11.718671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:11.719622 (+ 951us) server_negotiation.cc:407] Connection header received
0808 02:25:11.719866 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:11.719870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:11.719929 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:11.720058 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:11.720719 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.721425 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.723620 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.724586 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.724781 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.724890 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:11.725426 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 02:25:11.725559 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:25:11.726074 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.718330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46063 (local address 127.19.231.3:58529)
0808 02:25:11.719458 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:11.719479 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:11.719699 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:11.720072 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:11.720076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:11.720100 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:11.720576 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.720585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.721584 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.721594 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.723441 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:11.723456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.724851 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.724876 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:11.725154 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.725179 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 02:25:11.725391 (+ 212us) client_negotiation.cc:240] Negotiation successful
0808 02:25:11.725531 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":1,"thread_start_us":541,"threads_started":1}
I20260808 02:25:11.727037 28669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaed749af29461bab471e208581afa9" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c955d9ea567d48b4b04be23fa4800296" is_pre_election: true
I20260808 02:25:11.727068 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaed749af29461bab471e208581afa9" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" is_pre_election: true
I20260808 02:25:11.727707 28669 raft_consensus.cc:2454] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 0.
I20260808 02:25:11.727707 28522 raft_consensus.cc:2454] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 0.
I20260808 02:25:11.728405 28742 leader_election.cc:302] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e68e71d9d38456a9bf6d4e4887db304, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:11.728856 28896 raft_consensus.cc:2781] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:11.729040 28896 raft_consensus.cc:482] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:11.729152 28896 raft_consensus.cc:3037] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:11.731680 28896 raft_consensus.cc:504] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.732342 28896 leader_election.cc:288] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 election: Requested vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911)
I20260808 02:25:11.732697 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaed749af29461bab471e208581afa9" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f"
I20260808 02:25:11.732864 28522 raft_consensus.cc:3037] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:11.732882 28669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaed749af29461bab471e208581afa9" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c955d9ea567d48b4b04be23fa4800296"
I20260808 02:25:11.733114 28669 raft_consensus.cc:3037] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:11.735707 28669 raft_consensus.cc:2454] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 1.
I20260808 02:25:11.735735 28522 raft_consensus.cc:2454] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 1.
I20260808 02:25:11.736148 28741 leader_election.cc:302] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, 7e68e71d9d38456a9bf6d4e4887db304; no voters:
I20260808 02:25:11.736481 28896 raft_consensus.cc:2781] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:11.737025 28896 raft_consensus.cc:686] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 LEADER]: Becoming Leader. State: Replica: 7e68e71d9d38456a9bf6d4e4887db304, State: Running, Role: LEADER
I20260808 02:25:11.737555 28896 consensus_queue.cc:237] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } }
I20260808 02:25:11.742566 28361 catalog_manager.cc:5495] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "7e68e71d9d38456a9bf6d4e4887db304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } health_report { overall_health: UNKNOWN } } }
I20260808 02:25:11.812542 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:11.806957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.19.231.3:41513)
0808 02:25:11.807190 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:11.807196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:11.807217 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:25:11.807473 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:11.807479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:11.807598 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:11.807762 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:11.808182 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.808933 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.810319 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:11.810848 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:11.810967 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:11.811400 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:11.811970 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:11.812108 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:11.812223 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 02:25:11.812309 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:25:11.821120 28669 raft_consensus.cc:1261] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Refusing update from remote peer 7e68e71d9d38456a9bf6d4e4887db304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:11.821202 28522 raft_consensus.cc:1261] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Refusing update from remote peer 7e68e71d9d38456a9bf6d4e4887db304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:11.822336 28896 consensus_queue.cc:1035] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:11.822686 28901 consensus_queue.cc:1035] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:11.858752 28360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:11.859086 28360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:11.863137 28360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Sending DeleteTablet for 3 replicas of tablet ecaed749af29461bab471e208581afa9
I20260808 02:25:11.864291 28792 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaed749af29461bab471e208581afa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:11 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:35712
I20260808 02:25:11.864405 28502 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaed749af29461bab471e208581afa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:11 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43750
I20260808 02:25:11.864733 28648 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaed749af29461bab471e208581afa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:11 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:41916
I20260808 02:25:11.866920 28360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:11.868613 28360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:11.877146 28914 tablet_replica.cc:330] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: stopping tablet replica
I20260808 02:25:11.877524 28916 tablet_replica.cc:330] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: stopping tablet replica
I20260808 02:25:11.877848 28915 tablet_replica.cc:330] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: stopping tablet replica
I20260808 02:25:11.878175 28916 raft_consensus.cc:2229] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:11.878806 28916 raft_consensus.cc:2258] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:11.878772 28914 raft_consensus.cc:2229] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:11.879182 28915 raft_consensus.cc:2229] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:11.879784 28915 raft_consensus.cc:2258] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:11.880617 28914 raft_consensus.cc:2258] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:11.881172 28648 tablet_service.cc:1467] Processing CreateTablet for tablet 24bfd6b977c7464db0c2e7d776990b9c (DEFAULT_TABLE table=test-table [id=afd681ae75c64d1c84cc3b2fa0c639b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:11.881464 28792 tablet_service.cc:1467] Processing CreateTablet for tablet 24bfd6b977c7464db0c2e7d776990b9c (DEFAULT_TABLE table=test-table [id=afd681ae75c64d1c84cc3b2fa0c639b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:11.881701 28648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bfd6b977c7464db0c2e7d776990b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:11.882086 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bfd6b977c7464db0c2e7d776990b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:11.882225 28501 tablet_service.cc:1467] Processing CreateTablet for tablet 24bfd6b977c7464db0c2e7d776990b9c (DEFAULT_TABLE table=test-table [id=afd681ae75c64d1c84cc3b2fa0c639b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:11.882668 28501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24bfd6b977c7464db0c2e7d776990b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:11.888731 28892 tablet_bootstrap.cc:492] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: Bootstrap starting.
I20260808 02:25:11.888808 28890 tablet_bootstrap.cc:492] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: Bootstrap starting.
I20260808 02:25:11.889458 28915 ts_tablet_manager.cc:1905] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:11.890633 28892 tablet_bootstrap.cc:654] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:11.890878 28890 tablet_bootstrap.cc:654] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:11.893594 28891 tablet_bootstrap.cc:492] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: Bootstrap starting.
I20260808 02:25:11.895380 28914 ts_tablet_manager.cc:1905] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:11.895455 28891 tablet_bootstrap.cc:654] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:11.900480 28891 tablet_bootstrap.cc:492] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: No bootstrap required, opened a new log
I20260808 02:25:11.900645 28891 ts_tablet_manager.cc:1397] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 02:25:11.901454 28891 raft_consensus.cc:348] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:11.901715 28891 raft_consensus.cc:374] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:11.901818 28891 raft_consensus.cc:729] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179ff45c551e4d5b8d4dd88f09ab4e3f, State: Initialized, Role: FOLLOWER
I20260808 02:25:11.902103 28891 consensus_queue.cc:260] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:11.887145 28916 ts_tablet_manager.cc:1905] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:11.905089 28891 ts_tablet_manager.cc:1428] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260808 02:25:11.907538 28890 tablet_bootstrap.cc:492] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: No bootstrap required, opened a new log
I20260808 02:25:11.907770 28890 ts_tablet_manager.cc:1397] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.012s
I20260808 02:25:11.908677 28892 tablet_bootstrap.cc:492] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: No bootstrap required, opened a new log
I20260808 02:25:11.908908 28892 ts_tablet_manager.cc:1397] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20260808 02:25:11.908797 28890 raft_consensus.cc:348] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:11.909158 28890 raft_consensus.cc:374] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:11.909339 28890 raft_consensus.cc:729] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e68e71d9d38456a9bf6d4e4887db304, State: Initialized, Role: FOLLOWER
I20260808 02:25:11.909771 28890 consensus_queue.cc:260] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:11.909945 28892 raft_consensus.cc:348] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:11.910213 28892 raft_consensus.cc:374] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:11.910338 28892 raft_consensus.cc:729] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c955d9ea567d48b4b04be23fa4800296, State: Initialized, Role: FOLLOWER
I20260808 02:25:11.910684 28892 consensus_queue.cc:260] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:11.911419 28892 ts_tablet_manager.cc:1428] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:11.911652 28916 ts_tablet_manager.cc:1918] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:11.911852 28916 log.cc:1196] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/ecaed749af29461bab471e208581afa9
I20260808 02:25:11.912504 28916 ts_tablet_manager.cc:1939] T ecaed749af29461bab471e208581afa9 P c955d9ea567d48b4b04be23fa4800296: Deleting consensus metadata
I20260808 02:25:11.912946 28890 ts_tablet_manager.cc:1428] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260808 02:25:11.914642 28345 catalog_manager.cc:4841] TS c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911): tablet ecaed749af29461bab471e208581afa9 (table test-table [id=9eb7f84ebd94486faa1edebf6d738f9b]) successfully deleted
I20260808 02:25:11.917585 28914 ts_tablet_manager.cc:1918] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:11.917801 28914 log.cc:1196] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/ecaed749af29461bab471e208581afa9
I20260808 02:25:11.917948 28915 ts_tablet_manager.cc:1918] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:11.918174 28915 log.cc:1196] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/ecaed749af29461bab471e208581afa9
I20260808 02:25:11.918471 28914 ts_tablet_manager.cc:1939] T ecaed749af29461bab471e208581afa9 P 7e68e71d9d38456a9bf6d4e4887db304: Deleting consensus metadata
I20260808 02:25:11.918856 28915 ts_tablet_manager.cc:1939] T ecaed749af29461bab471e208581afa9 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting consensus metadata
I20260808 02:25:11.920491 28346 catalog_manager.cc:4841] TS 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513): tablet ecaed749af29461bab471e208581afa9 (table test-table [id=9eb7f84ebd94486faa1edebf6d738f9b]) successfully deleted
I20260808 02:25:11.920672 28344 catalog_manager.cc:4841] TS 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063): tablet ecaed749af29461bab471e208581afa9 (table test-table [id=9eb7f84ebd94486faa1edebf6d738f9b]) successfully deleted
I20260808 02:25:12.149909 28898 raft_consensus.cc:482] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:12.150219 28898 raft_consensus.cc:504] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.152546 28898 leader_election.cc:288] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513)
I20260808 02:25:12.161866 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.153313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:38077 (local address 127.19.231.1:46063)
0808 02:25:12.153534 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.153543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.154067 (+ 524us) server_negotiation.cc:407] Connection header received
0808 02:25:12.154456 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.154462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.154542 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.154645 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:12.155431 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.156621 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.159169 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.160464 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.160613 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.160686 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.161433 (+ 747us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.161586 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:25:12.161891 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.153149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:35721 (local address 127.19.231.3:41513)
0808 02:25:12.153367 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.153374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.154418 (+ 1044us) server_negotiation.cc:407] Connection header received
0808 02:25:12.154736 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.154745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.154837 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.154966 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:12.155618 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.156421 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.159185 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.160487 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.160609 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.160686 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.161434 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.161572 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 02:25:12.162698 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.152909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:41513 (local address 127.19.231.2:35721)
0808 02:25:12.154284 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.154298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:12.154516 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:12.154997 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:12.155004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:12.155036 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:12.155469 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.155478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.156570 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.156577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.159007 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.159016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.160910 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.160918 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.161199 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.161254 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:25:12.162142 (+ 888us) client_negotiation.cc:240] Negotiation successful
0808 02:25:12.162238 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"mutex_wait_us":124,"thread_start_us":658,"threads_started":1}
I20260808 02:25:12.162725 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.152851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46063 (local address 127.19.231.2:38077)
0808 02:25:12.153915 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.153939 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:12.154236 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:12.154703 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:12.154710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:12.154734 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:12.155197 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.155210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.156761 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.156769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.158975 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.158996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.160948 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.160953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.161204 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.161256 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 02:25:12.162057 (+ 801us) client_negotiation.cc:240] Negotiation successful
0808 02:25:12.162201 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":92,"thread_start_us":1168,"threads_started":1}
I20260808 02:25:12.163374 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfd6b977c7464db0c2e7d776990b9c" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" is_pre_election: true
I20260808 02:25:12.163580 28522 raft_consensus.cc:2454] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 0.
I20260808 02:25:12.164042 28596 leader_election.cc:302] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:12.164409 28898 raft_consensus.cc:2781] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:12.164507 28898 raft_consensus.cc:482] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:12.164589 28898 raft_consensus.cc:3037] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.166831 28898 raft_consensus.cc:504] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.167589 28898 leader_election.cc:288] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 election: Requested vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513)
I20260808 02:25:12.167418 28815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfd6b977c7464db0c2e7d776990b9c" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e68e71d9d38456a9bf6d4e4887db304" is_pre_election: true
I20260808 02:25:12.167929 28815 raft_consensus.cc:2454] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 0.
I20260808 02:25:12.169130 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfd6b977c7464db0c2e7d776990b9c" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f"
I20260808 02:25:12.169340 28522 raft_consensus.cc:3037] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.171518 28522 raft_consensus.cc:2454] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 1.
I20260808 02:25:12.171970 28596 leader_election.cc:302] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:12.171949 28815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24bfd6b977c7464db0c2e7d776990b9c" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e68e71d9d38456a9bf6d4e4887db304"
I20260808 02:25:12.172197 28815 raft_consensus.cc:3037] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.172441 28898 raft_consensus.cc:2781] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:12.175709 28898 raft_consensus.cc:686] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 1 LEADER]: Becoming Leader. State: Replica: c955d9ea567d48b4b04be23fa4800296, State: Running, Role: LEADER
I20260808 02:25:12.176204 28898 consensus_queue.cc:237] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.176578 28815 raft_consensus.cc:2454] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 1.
I20260808 02:25:12.180299 28360 catalog_manager.cc:5495] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 reported cstate change: term changed from 0 to 1, leader changed from <none> to c955d9ea567d48b4b04be23fa4800296 (127.19.231.2). New cstate: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } health_report { overall_health: UNKNOWN } } }
Aug 08 02:25:12 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.62@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:12.236544 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.213236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.19.231.62:39901)
0808 02:25:12.215953 (+ 2717us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.215959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.215986 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:25:12.216068 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.216076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.216155 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.216279 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:12.217496 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.218837 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.220130 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.220837 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.220991 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.229834 (+ 8843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:12.229839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:12.229843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:12.229876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:12.232864 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:12.233945 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:12.233949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:12.233952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:12.234144 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:12.234693 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:12.234704 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:12.234706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:12.235118 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:12.235272 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.235846 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.236217 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":179,"server-negotiator.queue_time_us":2586,"thread_start_us":2445,"threads_started":1}
I20260808 02:25:12.236647 28929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.213183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39901 (local address 127.0.0.1:60554)
0808 02:25:12.215231 (+ 2048us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.215262 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:12.215433 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:12.216340 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:12.216344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:12.217024 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:12.217274 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.217283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.218988 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.218992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.219971 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.219982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.221019 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.221024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.221155 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.226325 (+ 5170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:12.226364 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:12.229554 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:12.233032 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:12.233049 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:12.233058 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:12.233813 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:12.234279 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:12.234286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:12.234291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:12.234500 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:12.235291 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:12.235302 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:12.235516 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 02:25:12.235744 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 02:25:12.236084 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1853,"thread_start_us":1655,"threads_started":1}
I20260808 02:25:12.242444 28360 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:60554:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 02:25:12.243764 28360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:12.252152 28501 tablet_service.cc:1467] Processing CreateTablet for tablet 64d70cf199704bdb8c6a6e98e6017744 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4943a9f79544482d8ddc5b4ae301ea3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:25:12.252735 28501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d70cf199704bdb8c6a6e98e6017744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.252807 28648 tablet_service.cc:1467] Processing CreateTablet for tablet 64d70cf199704bdb8c6a6e98e6017744 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4943a9f79544482d8ddc5b4ae301ea3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:25:12.253453 28648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d70cf199704bdb8c6a6e98e6017744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.258128 28891 tablet_bootstrap.cc:492] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Bootstrap starting.
I20260808 02:25:12.258818 28792 tablet_service.cc:1467] Processing CreateTablet for tablet 64d70cf199704bdb8c6a6e98e6017744 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4943a9f79544482d8ddc5b4ae301ea3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 02:25:12.259410 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d70cf199704bdb8c6a6e98e6017744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.259948 28891 tablet_bootstrap.cc:654] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.262101 28892 tablet_bootstrap.cc:492] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: Bootstrap starting.
I20260808 02:25:12.264073 28892 tablet_bootstrap.cc:654] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.269189 28890 tablet_bootstrap.cc:492] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: Bootstrap starting.
I20260808 02:25:12.271591 28890 tablet_bootstrap.cc:654] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.273402 28892 tablet_bootstrap.cc:492] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: No bootstrap required, opened a new log
I20260808 02:25:12.273658 28892 ts_tablet_manager.cc:1397] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260808 02:25:12.274650 28892 raft_consensus.cc:348] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.275055 28892 raft_consensus.cc:374] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.275197 28892 raft_consensus.cc:729] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c955d9ea567d48b4b04be23fa4800296, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.275579 28892 consensus_queue.cc:260] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.276165 28898 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.276505 28898 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:25:12.277084 28890 tablet_bootstrap.cc:492] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: No bootstrap required, opened a new log
I20260808 02:25:12.277277 28890 ts_tablet_manager.cc:1397] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260808 02:25:12.278139 28890 raft_consensus.cc:348] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.278378 28890 raft_consensus.cc:374] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.278467 28890 raft_consensus.cc:729] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e68e71d9d38456a9bf6d4e4887db304, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.278707 28891 tablet_bootstrap.cc:492] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: No bootstrap required, opened a new log
I20260808 02:25:12.278913 28891 ts_tablet_manager.cc:1397] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.012s
I20260808 02:25:12.278954 28890 consensus_queue.cc:260] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.279210 28892 ts_tablet_manager.cc:1428] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 02:25:12.279587 28896 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.279870 28896 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:25:12.279917 28891 raft_consensus.cc:348] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.280161 28891 raft_consensus.cc:374] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.280243 28891 raft_consensus.cc:729] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179ff45c551e4d5b8d4dd88f09ab4e3f, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.280257 28890 ts_tablet_manager.cc:1428] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 02:25:12.280500 28891 consensus_queue.cc:260] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.281134 28891 ts_tablet_manager.cc:1428] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:12.282039 28897 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.282403 28897 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:25:12.301893 28898 raft_consensus.cc:482] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:12.302183 28898 raft_consensus.cc:504] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.303007 28898 leader_election.cc:288] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513)
I20260808 02:25:12.303624 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d70cf199704bdb8c6a6e98e6017744" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" is_pre_election: true
I20260808 02:25:12.303670 28815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d70cf199704bdb8c6a6e98e6017744" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e68e71d9d38456a9bf6d4e4887db304" is_pre_election: true
I20260808 02:25:12.303917 28522 raft_consensus.cc:2454] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 0.
I20260808 02:25:12.304016 28815 raft_consensus.cc:2454] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 0.
I20260808 02:25:12.304461 28596 leader_election.cc:302] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:12.304775 28898 raft_consensus.cc:2781] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:12.304893 28898 raft_consensus.cc:482] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:12.305081 28898 raft_consensus.cc:3037] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.307808 28898 raft_consensus.cc:504] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.308423 28898 leader_election.cc:288] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 election: Requested vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513)
I20260808 02:25:12.308861 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d70cf199704bdb8c6a6e98e6017744" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f"
I20260808 02:25:12.308993 28815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d70cf199704bdb8c6a6e98e6017744" candidate_uuid: "c955d9ea567d48b4b04be23fa4800296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e68e71d9d38456a9bf6d4e4887db304"
I20260808 02:25:12.309108 28522 raft_consensus.cc:3037] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.309262 28815 raft_consensus.cc:3037] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.311542 28522 raft_consensus.cc:2454] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 1.
I20260808 02:25:12.311987 28596 leader_election.cc:302] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:12.312331 28898 raft_consensus.cc:2781] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:12.312551 28898 raft_consensus.cc:686] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [term 1 LEADER]: Becoming Leader. State: Replica: c955d9ea567d48b4b04be23fa4800296, State: Running, Role: LEADER
I20260808 02:25:12.312520 28815 raft_consensus.cc:2454] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c955d9ea567d48b4b04be23fa4800296 in term 1.
I20260808 02:25:12.312876 28898 consensus_queue.cc:237] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.314847 28922 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: TxnStatusTablet state changed. Reason: New leader c955d9ea567d48b4b04be23fa4800296. Latest consensus state: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.315219 28922 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:25:12.316431 28935 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 02:25:12.316747 28935 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 02:25:12.317238 28360 catalog_manager.cc:5495] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 reported cstate change: term changed from 0 to 1, leader changed from <none> to c955d9ea567d48b4b04be23fa4800296 (127.19.231.2). New cstate: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260808 02:25:12.350013 28929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.338999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:38911 (local address 127.0.0.1:41928)
0808 02:25:12.339270 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.339295 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:12.339454 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:12.343592 (+ 4138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:12.343601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:12.343621 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:25:12.343843 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.343864 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.345491 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.345498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.346898 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.346913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.347934 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.347941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.348129 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.348249 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:25:12.349287 (+ 1038us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:25:12.349317 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:25:12.349405 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 02:25:12.349661 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:25:12.350046 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.339587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.19.231.2:38911)
0808 02:25:12.343132 (+ 3545us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.343137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.343164 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:25:12.343259 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.343263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.343350 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.343547 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:12.344012 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.345298 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.347050 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.347749 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.347918 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.348543 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:12.349142 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:12.349435 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.349511 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.349619 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3421,"thread_start_us":3223,"threads_started":1}
I20260808 02:25:12.355162 28522 raft_consensus.cc:1261] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Refusing update from remote peer c955d9ea567d48b4b04be23fa4800296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:12.355572 28815 raft_consensus.cc:1261] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Refusing update from remote peer c955d9ea567d48b4b04be23fa4800296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:12.356215 28922 consensus_queue.cc:1035] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:12.356617 28898 consensus_queue.cc:1035] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:12.364143 28897 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: TxnStatusTablet state changed. Reason: New leader c955d9ea567d48b4b04be23fa4800296. Latest consensus state: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.364468 28897 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:25:12.372378 28939 mvcc.cc:205] Tried to move back new op lower bound from 7316094616998969344 to 7316094616836575232. Current Snapshot: MvccSnapshot[applied={T|T < 7316094616998969344}]
I20260808 02:25:12.378264 28922 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.378582 28922 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:25:12.383170 28897 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.383461 28897 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P 179ff45c551e4d5b8d4dd88f09ab4e3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:25:12.385843 28896 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: TxnStatusTablet state changed. Reason: New leader c955d9ea567d48b4b04be23fa4800296. Latest consensus state: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.386189 28896 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:25:12.392467 28898 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.392791 28898 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P c955d9ea567d48b4b04be23fa4800296: This TxnStatusTablet replica's current role is: LEADER
I20260808 02:25:12.408857 28896 tablet_replica.cc:439] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c955d9ea567d48b4b04be23fa4800296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } } }
I20260808 02:25:12.409163 28896 tablet_replica.cc:442] T 64d70cf199704bdb8c6a6e98e6017744 P 7e68e71d9d38456a9bf6d4e4887db304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 02:25:12.421630 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.413652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.19.231.2:38911)
0808 02:25:12.413847 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.413855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.414373 (+ 518us) server_negotiation.cc:407] Connection header received
0808 02:25:12.414727 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.414733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.414819 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.414892 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:12.415910 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.417055 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.419190 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.419769 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.419869 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.420524 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:12.420658 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:12.420731 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.421223 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.421318 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 02:25:12.457229 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.439154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:58643 (local address 127.19.231.2:38911)
0808 02:25:12.439572 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.439585 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.451347 (+ 11762us) server_negotiation.cc:407] Connection header received
0808 02:25:12.451573 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.451578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.451648 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.451753 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:12.452482 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.453283 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.455420 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.455834 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.455975 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.456472 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:12.456570 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:12.456665 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.456773 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.456842 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260808 02:25:12.457305 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.439106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:38911 (local address 127.19.231.2:58643)
0808 02:25:12.451234 (+ 12128us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.451253 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:12.451437 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:12.451902 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:12.451907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:12.451918 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:25:12.452298 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.452306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.453414 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.453421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.455269 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.455283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.456034 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.456041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.456217 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.456304 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:25:12.456663 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:25:12.456675 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 02:25:12.456756 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 02:25:12.456883 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11919,"thread_start_us":11764,"threads_started":1}
I20260808 02:25:12.467834 28522 raft_consensus.cc:1261] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Refusing update from remote peer c955d9ea567d48b4b04be23fa4800296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:12.467990 28815 raft_consensus.cc:1261] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Refusing update from remote peer c955d9ea567d48b4b04be23fa4800296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:12.468411 28922 consensus_queue.cc:1035] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:12.468736 28898 consensus_queue.cc:1035] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:12.538709 28359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:12.538981 28359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:12.542172 28359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Sending DeleteTablet for 3 replicas of tablet 24bfd6b977c7464db0c2e7d776990b9c
I20260808 02:25:12.543031 28648 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bfd6b977c7464db0c2e7d776990b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:12 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:41916
I20260808 02:25:12.543081 28501 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bfd6b977c7464db0c2e7d776990b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:12 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43750
I20260808 02:25:12.543320 28792 tablet_service.cc:1514] Processing DeleteTablet for tablet 24bfd6b977c7464db0c2e7d776990b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:12 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:35712
I20260808 02:25:12.556351 28960 tablet_replica.cc:330] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: stopping tablet replica
I20260808 02:25:12.557324 28960 raft_consensus.cc:2229] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:12.557999 28960 raft_consensus.cc:2258] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:12.558223 28959 tablet_replica.cc:330] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: stopping tablet replica
I20260808 02:25:12.558715 28958 tablet_replica.cc:330] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: stopping tablet replica
I20260808 02:25:12.558995 28958 raft_consensus.cc:2229] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:12.559327 28959 raft_consensus.cc:2229] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:12.559473 28960 ts_tablet_manager.cc:1905] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:12.560146 28959 raft_consensus.cc:2258] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:12.560436 28958 raft_consensus.cc:2258] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:12.561966 28958 ts_tablet_manager.cc:1905] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:12.563109 28959 ts_tablet_manager.cc:1905] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:12.566856 28960 ts_tablet_manager.cc:1918] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:25:12.567044 28960 log.cc:1196] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/24bfd6b977c7464db0c2e7d776990b9c
I20260808 02:25:12.567592 28960 ts_tablet_manager.cc:1939] T 24bfd6b977c7464db0c2e7d776990b9c P 7e68e71d9d38456a9bf6d4e4887db304: Deleting consensus metadata
I20260808 02:25:12.568799 28346 catalog_manager.cc:4841] TS 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513): tablet 24bfd6b977c7464db0c2e7d776990b9c (table test-table [id=afd681ae75c64d1c84cc3b2fa0c639b2]) successfully deleted
I20260808 02:25:12.572185 28958 ts_tablet_manager.cc:1918] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:25:12.572444 28958 log.cc:1196] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/24bfd6b977c7464db0c2e7d776990b9c
I20260808 02:25:12.573089 28958 ts_tablet_manager.cc:1939] T 24bfd6b977c7464db0c2e7d776990b9c P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting consensus metadata
I20260808 02:25:12.573714 28959 ts_tablet_manager.cc:1918] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:25:12.573906 28959 log.cc:1196] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/24bfd6b977c7464db0c2e7d776990b9c
I20260808 02:25:12.574355 28344 catalog_manager.cc:4841] TS 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063): tablet 24bfd6b977c7464db0c2e7d776990b9c (table test-table [id=afd681ae75c64d1c84cc3b2fa0c639b2]) successfully deleted
I20260808 02:25:12.574433 28959 ts_tablet_manager.cc:1939] T 24bfd6b977c7464db0c2e7d776990b9c P c955d9ea567d48b4b04be23fa4800296: Deleting consensus metadata
I20260808 02:25:12.575652 28345 catalog_manager.cc:4841] TS c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911): tablet 24bfd6b977c7464db0c2e7d776990b9c (table test-table [id=afd681ae75c64d1c84cc3b2fa0c639b2]) successfully deleted
Aug 08 02:25:12 dist-test-slave-pgkr krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 02:25:12 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:12 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:25:12.628904 28930 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:12.613465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.19.231.62:39901)
0808 02:25:12.613682 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.613689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.614293 (+ 604us) server_negotiation.cc:407] Connection header received
0808 02:25:12.614636 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.614644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.614743 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.614873 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:12.616811 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.618064 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.620175 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.620936 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.621092 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.628613 (+ 7521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60556: BlockingRecv error: recv got EOF from 127.0.0.1:60556 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 02:25:12 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM
I20260808 02:25:12.667354 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.644723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.19.231.62:39901)
0808 02:25:12.644949 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.644956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.649664 (+ 4708us) server_negotiation.cc:407] Connection header received
0808 02:25:12.649881 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.649887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.649988 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.650128 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:12.652062 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.653330 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.654991 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.655860 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.656019 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.660392 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:12.660399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:12.660405 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:12.660458 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:12.664266 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:12.664952 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:12.664958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:12.664962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:12.665183 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:12.665553 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:12.665559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:12.665563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:12.665951 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:12.666253 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.666545 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.666879 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:25:12.671626 28359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:12.672415 28359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:12.679641 28501 tablet_service.cc:1467] Processing CreateTablet for tablet f8738da32a38460c8743b2df782e0861 (DEFAULT_TABLE table=test-table [id=eea5be4756df4b95949c1c888776c1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:12.679950 28792 tablet_service.cc:1467] Processing CreateTablet for tablet f8738da32a38460c8743b2df782e0861 (DEFAULT_TABLE table=test-table [id=eea5be4756df4b95949c1c888776c1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:12.680124 28501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8738da32a38460c8743b2df782e0861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.680454 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8738da32a38460c8743b2df782e0861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.685832 28891 tablet_bootstrap.cc:492] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Bootstrap starting.
I20260808 02:25:12.685941 28648 tablet_service.cc:1467] Processing CreateTablet for tablet f8738da32a38460c8743b2df782e0861 (DEFAULT_TABLE table=test-table [id=eea5be4756df4b95949c1c888776c1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:12.686426 28648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8738da32a38460c8743b2df782e0861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.687614 28891 tablet_bootstrap.cc:654] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.687781 28890 tablet_bootstrap.cc:492] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: Bootstrap starting.
I20260808 02:25:12.689502 28890 tablet_bootstrap.cc:654] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.696417 28892 tablet_bootstrap.cc:492] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: Bootstrap starting.
I20260808 02:25:12.696725 28891 tablet_bootstrap.cc:492] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: No bootstrap required, opened a new log
I20260808 02:25:12.696942 28891 ts_tablet_manager.cc:1397] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260808 02:25:12.697831 28891 raft_consensus.cc:348] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.698189 28892 tablet_bootstrap.cc:654] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.698280 28891 raft_consensus.cc:374] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.698449 28891 raft_consensus.cc:729] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179ff45c551e4d5b8d4dd88f09ab4e3f, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.698822 28891 consensus_queue.cc:260] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.702852 28891 ts_tablet_manager.cc:1428] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260808 02:25:12.703677 28890 tablet_bootstrap.cc:492] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: No bootstrap required, opened a new log
I20260808 02:25:12.703898 28890 ts_tablet_manager.cc:1397] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20260808 02:25:12.704851 28890 raft_consensus.cc:348] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.705199 28890 raft_consensus.cc:374] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.705335 28890 raft_consensus.cc:729] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e68e71d9d38456a9bf6d4e4887db304, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.705638 28890 consensus_queue.cc:260] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.706314 28890 ts_tablet_manager.cc:1428] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 02:25:12.709692 28892 tablet_bootstrap.cc:492] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: No bootstrap required, opened a new log
I20260808 02:25:12.709873 28892 ts_tablet_manager.cc:1397] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260808 02:25:12.710839 28892 raft_consensus.cc:348] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.711081 28892 raft_consensus.cc:374] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.711225 28892 raft_consensus.cc:729] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c955d9ea567d48b4b04be23fa4800296, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.711472 28892 consensus_queue.cc:260] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.711983 28892 ts_tablet_manager.cc:1428] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:12.780220 28897 raft_consensus.cc:482] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:12.780481 28897 raft_consensus.cc:504] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.782536 28897 leader_election.cc:288] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911), 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513)
I20260808 02:25:12.791473 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.783237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:35783 (local address 127.19.231.2:38911)
0808 02:25:12.783454 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.783461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.783978 (+ 517us) server_negotiation.cc:407] Connection header received
0808 02:25:12.784294 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.784300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.784380 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.784445 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:12.785221 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.786503 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.789242 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.790395 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.790635 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.790732 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.791007 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.791145 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:25:12.791633 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.782786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:38911 (local address 127.19.231.1:35783)
0808 02:25:12.783865 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.783885 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:12.784114 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:12.784667 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:12.784672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:12.784698 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:12.784972 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.784985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.786687 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.786692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.789019 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.789035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.790574 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.790584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.790819 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.790858 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:25:12.790978 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 02:25:12.791139 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":34,"thread_start_us":805,"threads_started":1}
I20260808 02:25:12.792322 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.783353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:41513 (local address 127.19.231.1:50697)
0808 02:25:12.784299 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:12.784323 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:12.784467 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:12.785475 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:12.785481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:12.785507 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:12.785888 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.785899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.787350 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.787357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.789433 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:12.789449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.791188 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.791195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:12.791368 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.791403 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:25:12.791475 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 02:25:12.791609 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":31,"thread_start_us":570,"threads_started":1}
I20260808 02:25:12.792604 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.782935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:50697 (local address 127.19.231.3:41513)
0808 02:25:12.785085 (+ 2150us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.785095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.785124 (+ 29us) server_negotiation.cc:407] Connection header received
0808 02:25:12.785213 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.785218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.785319 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.785447 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:12.786065 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.787194 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.789618 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.790917 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.791946 (+ 1029us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.792027 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.792093 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.792240 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1885,"thread_start_us":1777,"threads_started":1}
I20260808 02:25:12.792654 28669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8738da32a38460c8743b2df782e0861" candidate_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c955d9ea567d48b4b04be23fa4800296" is_pre_election: true
I20260808 02:25:12.792951 28669 raft_consensus.cc:2454] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179ff45c551e4d5b8d4dd88f09ab4e3f in term 0.
I20260808 02:25:12.793291 28815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8738da32a38460c8743b2df782e0861" candidate_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e68e71d9d38456a9bf6d4e4887db304" is_pre_election: true
I20260808 02:25:12.793540 28815 raft_consensus.cc:2454] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179ff45c551e4d5b8d4dd88f09ab4e3f in term 0.
I20260808 02:25:12.793598 28451 leader_election.cc:302] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:12.794114 28897 raft_consensus.cc:2781] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:12.794262 28897 raft_consensus.cc:482] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:12.794376 28897 raft_consensus.cc:3037] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.797691 28897 raft_consensus.cc:504] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.798907 28669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8738da32a38460c8743b2df782e0861" candidate_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c955d9ea567d48b4b04be23fa4800296"
I20260808 02:25:12.799178 28669 raft_consensus.cc:3037] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.799459 28815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8738da32a38460c8743b2df782e0861" candidate_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e68e71d9d38456a9bf6d4e4887db304"
I20260808 02:25:12.799682 28815 raft_consensus.cc:3037] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.801497 28669 raft_consensus.cc:2454] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179ff45c551e4d5b8d4dd88f09ab4e3f in term 1.
I20260808 02:25:12.801981 28451 leader_election.cc:302] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:12.802718 28897 leader_election.cc:288] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [CANDIDATE]: Term 1 election: Requested vote from peers c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911), 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513)
I20260808 02:25:12.802781 28815 raft_consensus.cc:2454] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179ff45c551e4d5b8d4dd88f09ab4e3f in term 1.
I20260808 02:25:12.802779 28979 raft_consensus.cc:2781] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:12.803282 28979 raft_consensus.cc:686] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 LEADER]: Becoming Leader. State: Replica: 179ff45c551e4d5b8d4dd88f09ab4e3f, State: Running, Role: LEADER
I20260808 02:25:12.803824 28979 consensus_queue.cc:237] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.808290 28360 catalog_manager.cc:5495] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1). New cstate: current_term: 1 leader_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260808 02:25:12.836419 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:12.829266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.19.231.1:46063)
0808 02:25:12.830124 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:12.830131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:12.830156 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:25:12.830266 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:12.830271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:12.830386 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:12.830741 (+ 355us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:12.831139 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.832334 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.833872 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:12.834486 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:12.834661 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:12.835154 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:12.835746 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:12.835909 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:12.835989 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 02:25:12.836071 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":526,"threads_started":1}
I20260808 02:25:12.840291 28815 raft_consensus.cc:1261] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Refusing update from remote peer 179ff45c551e4d5b8d4dd88f09ab4e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:12.840273 28669 raft_consensus.cc:1261] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Refusing update from remote peer 179ff45c551e4d5b8d4dd88f09ab4e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:12.841071 28979 consensus_queue.cc:1035] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:12.841401 28897 consensus_queue.cc:1035] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:12.845988 28940 mvcc.cc:205] Tried to move back new op lower bound from 7316094618986233856 to 7316094618849841152. Current Snapshot: MvccSnapshot[applied={T|T < 7316094618986233856}]
I20260808 02:25:12.852209 28941 mvcc.cc:205] Tried to move back new op lower bound from 7316094618986233856 to 7316094618849841152. Current Snapshot: MvccSnapshot[applied={T|T < 7316094618986233856 or (T in {7316094618986233856})}]
I20260808 02:25:12.864467 28359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:12.864745 28359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:12.868347 28359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Sending DeleteTablet for 3 replicas of tablet f8738da32a38460c8743b2df782e0861
I20260808 02:25:12.869450 28501 tablet_service.cc:1514] Processing DeleteTablet for tablet f8738da32a38460c8743b2df782e0861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:12 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43750
I20260808 02:25:12.869566 28648 tablet_service.cc:1514] Processing DeleteTablet for tablet f8738da32a38460c8743b2df782e0861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:12 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:41916
I20260808 02:25:12.869809 28958 tablet_replica.cc:330] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: stopping tablet replica
I20260808 02:25:12.869911 28792 tablet_service.cc:1514] Processing DeleteTablet for tablet f8738da32a38460c8743b2df782e0861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:12 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:35712
I20260808 02:25:12.870028 28958 raft_consensus.cc:2229] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:12.870200 28960 tablet_replica.cc:330] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: stopping tablet replica
I20260808 02:25:12.870313 28959 tablet_replica.cc:330] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: stopping tablet replica
I20260808 02:25:12.870409 28960 raft_consensus.cc:2229] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:12.870498 28959 raft_consensus.cc:2229] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:12.870671 28960 raft_consensus.cc:2258] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:12.870738 28959 raft_consensus.cc:2258] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:12.871797 28960 ts_tablet_manager.cc:1905] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:12.871768 28359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:12.872504 28359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:12.874256 28959 ts_tablet_manager.cc:1905] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:12.880926 28960 ts_tablet_manager.cc:1918] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:12.881191 28959 ts_tablet_manager.cc:1918] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:12.881417 28959 log.cc:1196] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/f8738da32a38460c8743b2df782e0861
I20260808 02:25:12.881925 28959 ts_tablet_manager.cc:1939] T f8738da32a38460c8743b2df782e0861 P c955d9ea567d48b4b04be23fa4800296: Deleting consensus metadata
I20260808 02:25:12.883061 28345 catalog_manager.cc:4841] TS c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911): tablet f8738da32a38460c8743b2df782e0861 (table test-table [id=eea5be4756df4b95949c1c888776c1f3]) successfully deleted
I20260808 02:25:12.886875 28501 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de (DEFAULT_TABLE table=test-table [id=6f9e1ff5374342f5804aa760f8fd33db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:12.887316 28958 raft_consensus.cc:2258] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:12.887387 28501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.887477 28648 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de (DEFAULT_TABLE table=test-table [id=6f9e1ff5374342f5804aa760f8fd33db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:12.887899 28648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.889350 28960 log.cc:1196] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/f8738da32a38460c8743b2df782e0861
I20260808 02:25:12.889817 28960 ts_tablet_manager.cc:1939] T f8738da32a38460c8743b2df782e0861 P 7e68e71d9d38456a9bf6d4e4887db304: Deleting consensus metadata
I20260808 02:25:12.890842 28346 catalog_manager.cc:4841] TS 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513): tablet f8738da32a38460c8743b2df782e0861 (table test-table [id=eea5be4756df4b95949c1c888776c1f3]) successfully deleted
I20260808 02:25:12.892030 28958 ts_tablet_manager.cc:1905] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:12.885883 28792 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de (DEFAULT_TABLE table=test-table [id=6f9e1ff5374342f5804aa760f8fd33db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:12.894927 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:12.895483 28892 tablet_bootstrap.cc:492] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: Bootstrap starting.
I20260808 02:25:12.897238 28892 tablet_bootstrap.cc:654] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.903199 28958 ts_tablet_manager.cc:1918] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:12.909919 28890 tablet_bootstrap.cc:492] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: Bootstrap starting.
I20260808 02:25:12.906833 28891 tablet_bootstrap.cc:492] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: Bootstrap starting.
I20260808 02:25:12.911830 28890 tablet_bootstrap.cc:654] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.912019 28891 tablet_bootstrap.cc:654] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:12.915084 28958 log.cc:1196] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/f8738da32a38460c8743b2df782e0861
I20260808 02:25:12.915787 28958 ts_tablet_manager.cc:1939] T f8738da32a38460c8743b2df782e0861 P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting consensus metadata
I20260808 02:25:12.918078 28344 catalog_manager.cc:4841] TS 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063): tablet f8738da32a38460c8743b2df782e0861 (table test-table [id=eea5be4756df4b95949c1c888776c1f3]) successfully deleted
I20260808 02:25:12.924034 28891 tablet_bootstrap.cc:492] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: No bootstrap required, opened a new log
I20260808 02:25:12.924386 28891 ts_tablet_manager.cc:1397] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260808 02:25:12.925745 28891 raft_consensus.cc:348] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.926232 28891 raft_consensus.cc:374] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.926393 28891 raft_consensus.cc:729] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179ff45c551e4d5b8d4dd88f09ab4e3f, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.926869 28891 consensus_queue.cc:260] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.928591 28891 ts_tablet_manager.cc:1428] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260808 02:25:12.931236 28892 tablet_bootstrap.cc:492] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: No bootstrap required, opened a new log
I20260808 02:25:12.931488 28892 ts_tablet_manager.cc:1397] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.013s
I20260808 02:25:12.931453 28890 tablet_bootstrap.cc:492] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: No bootstrap required, opened a new log
I20260808 02:25:12.932356 28892 raft_consensus.cc:348] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.932618 28890 ts_tablet_manager.cc:1397] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.012s
I20260808 02:25:12.932737 28892 raft_consensus.cc:374] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.932845 28892 raft_consensus.cc:729] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c955d9ea567d48b4b04be23fa4800296, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.933087 28892 consensus_queue.cc:260] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.933621 28890 raft_consensus.cc:348] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.933928 28892 ts_tablet_manager.cc:1428] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 02:25:12.933928 28890 raft_consensus.cc:374] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:12.934026 28890 raft_consensus.cc:729] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e68e71d9d38456a9bf6d4e4887db304, State: Initialized, Role: FOLLOWER
I20260808 02:25:12.934299 28890 consensus_queue.cc:260] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.936391 28890 ts_tablet_manager.cc:1428] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260808 02:25:12.977695 28896 raft_consensus.cc:482] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:12.977931 28896 raft_consensus.cc:504] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.978837 28896 leader_election.cc:288] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911)
I20260808 02:25:12.979449 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8d3e1722f45f8b3d7d2dbb98ad6de" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" is_pre_election: true
I20260808 02:25:12.979561 28669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8d3e1722f45f8b3d7d2dbb98ad6de" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c955d9ea567d48b4b04be23fa4800296" is_pre_election: true
I20260808 02:25:12.979735 28522 raft_consensus.cc:2454] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 0.
I20260808 02:25:12.979847 28669 raft_consensus.cc:2454] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 0.
I20260808 02:25:12.980387 28741 leader_election.cc:302] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179ff45c551e4d5b8d4dd88f09ab4e3f, 7e68e71d9d38456a9bf6d4e4887db304; no voters:
I20260808 02:25:12.980870 28896 raft_consensus.cc:2781] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:12.981014 28896 raft_consensus.cc:482] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:12.981127 28896 raft_consensus.cc:3037] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.984301 28896 raft_consensus.cc:504] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:12.985548 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8d3e1722f45f8b3d7d2dbb98ad6de" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f"
I20260808 02:25:12.985607 28669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc8d3e1722f45f8b3d7d2dbb98ad6de" candidate_uuid: "7e68e71d9d38456a9bf6d4e4887db304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c955d9ea567d48b4b04be23fa4800296"
I20260808 02:25:12.985806 28522 raft_consensus.cc:3037] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.985877 28669 raft_consensus.cc:3037] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:12.989339 28522 raft_consensus.cc:2454] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 1.
I20260808 02:25:12.989337 28669 raft_consensus.cc:2454] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e68e71d9d38456a9bf6d4e4887db304 in term 1.
I20260808 02:25:12.989833 28742 leader_election.cc:302] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e68e71d9d38456a9bf6d4e4887db304, c955d9ea567d48b4b04be23fa4800296; no voters:
I20260808 02:25:12.990235 28896 leader_election.cc:288] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [CANDIDATE]: Term 1 election: Requested vote from peers 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063), c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911)
I20260808 02:25:12.990492 28896 raft_consensus.cc:2781] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:12.990716 28896 raft_consensus.cc:686] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 LEADER]: Becoming Leader. State: Replica: 7e68e71d9d38456a9bf6d4e4887db304, State: Running, Role: LEADER
I20260808 02:25:12.991040 28896 consensus_queue.cc:237] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } }
I20260808 02:25:13.004693 28360 catalog_manager.cc:5495] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "7e68e71d9d38456a9bf6d4e4887db304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e68e71d9d38456a9bf6d4e4887db304" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41513 } health_report { overall_health: HEALTHY } } }
I20260808 02:25:13.044447 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.037731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.19.231.3:41513)
0808 02:25:13.038050 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:13.038057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:13.038356 (+ 299us) server_negotiation.cc:407] Connection header received
0808 02:25:13.038543 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:13.038551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:13.038688 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:13.038788 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:13.039237 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.040345 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.042272 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.043011 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.043178 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.043646 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:13.043783 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:13.043900 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:13.044031 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 02:25:13.044116 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260808 02:25:13.077849 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.069797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:39105 (local address 127.19.231.2:38911)
0808 02:25:13.070046 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:13.070056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:13.070592 (+ 536us) server_negotiation.cc:407] Connection header received
0808 02:25:13.070947 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:13.070953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:13.071041 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:13.071109 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:13.071944 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.073156 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.075405 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.076110 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.076205 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.076905 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:13.077066 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:13.077130 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:13.077390 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 02:25:13.077484 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:25:13.078744 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.069734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:38911 (local address 127.19.231.3:39105)
0808 02:25:13.070475 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:13.070495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:13.070749 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:13.071303 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:13.071308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:13.071322 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:25:13.071703 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:13.071714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.073410 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.073416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:13.075222 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:13.075239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.076396 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.076402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:13.076607 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.076689 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:25:13.077230 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:25:13.077248 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 02:25:13.078132 (+ 884us) client_negotiation.cc:240] Negotiation successful
0808 02:25:13.078281 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":397,"threads_started":1}
I20260808 02:25:13.096397 28669 raft_consensus.cc:1261] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Refusing update from remote peer 7e68e71d9d38456a9bf6d4e4887db304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:13.097340 28990 consensus_queue.cc:1035] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c955d9ea567d48b4b04be23fa4800296" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:13.099046 28522 raft_consensus.cc:1261] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Refusing update from remote peer 7e68e71d9d38456a9bf6d4e4887db304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:13.100488 28990 consensus_queue.cc:1035] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "179ff45c551e4d5b8d4dd88f09ab4e3f" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:13.165248 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.151507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:33913 (local address 127.19.231.3:41513)
0808 02:25:13.151712 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:13.151720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:13.152553 (+ 833us) server_negotiation.cc:407] Connection header received
0808 02:25:13.153279 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:13.153286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:13.153404 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:13.153464 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:13.154740 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.155566 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.157301 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.157961 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.158104 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.158849 (+ 745us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:13.158990 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:13.159081 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:13.164765 (+ 5684us) server_negotiation.cc:299] Negotiation successful
0808 02:25:13.164872 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:25:13.166355 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.151523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:41513 (local address 127.19.231.2:33913)
0808 02:25:13.152240 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:13.152469 (+ 229us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:13.153067 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:13.153959 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:13.153967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:13.153981 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 02:25:13.154506 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:13.154527 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.155728 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.155739 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:13.156945 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:13.156959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.158273 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.158281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:13.158498 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.158566 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 02:25:13.164592 (+ 6026us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 02:25:13.164606 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 02:25:13.165730 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0808 02:25:13.165898 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":375,"threads_started":1}
I20260808 02:25:13.237598 28360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:13.237860 28360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:13.240869 28360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 573a0de1e1cd4cfbb22266e7642a621f: Sending DeleteTablet for 3 replicas of tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de
I20260808 02:25:13.241743 28501 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:13 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:43750
I20260808 02:25:13.242076 28958 tablet_replica.cc:330] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: stopping tablet replica
I20260808 02:25:13.242043 28648 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:13 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:41916
I20260808 02:25:13.242271 28958 raft_consensus.cc:2229] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:13.242372 28959 tablet_replica.cc:330] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: stopping tablet replica
I20260808 02:25:13.242506 28958 raft_consensus.cc:2258] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:13.242565 28959 raft_consensus.cc:2229] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:13.242862 28959 raft_consensus.cc:2258] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:13.244038 28958 ts_tablet_manager.cc:1905] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:13.244123 28959 ts_tablet_manager.cc:1905] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:13.254462 28958 ts_tablet_manager.cc:1918] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:25:13.254719 28958 log.cc:1196] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/3bc8d3e1722f45f8b3d7d2dbb98ad6de
I20260808 02:25:13.255293 28958 ts_tablet_manager.cc:1939] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 179ff45c551e4d5b8d4dd88f09ab4e3f: Deleting consensus metadata
I20260808 02:25:13.256552 28344 catalog_manager.cc:4841] TS 179ff45c551e4d5b8d4dd88f09ab4e3f (127.19.231.1:46063): tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de (table test-table [id=6f9e1ff5374342f5804aa760f8fd33db]) successfully deleted
I20260808 02:25:13.257164 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 28433
I20260808 02:25:13.265420 28792 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:13 UTC) from {username='oryx', principal='oryx/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:35712
I20260808 02:25:13.265802 28960 tablet_replica.cc:330] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: stopping tablet replica
I20260808 02:25:13.266040 28960 raft_consensus.cc:2229] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:13.266495 28960 raft_consensus.cc:2258] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:13.269380 28960 ts_tablet_manager.cc:1905] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:13.272043 28959 ts_tablet_manager.cc:1918] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:25:13.272361 28959 log.cc:1196] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/3bc8d3e1722f45f8b3d7d2dbb98ad6de
I20260808 02:25:13.279006 28960 ts_tablet_manager.cc:1918] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 02:25:13.279284 28960 log.cc:1196] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/3bc8d3e1722f45f8b3d7d2dbb98ad6de
I20260808 02:25:13.279956 28960 ts_tablet_manager.cc:1939] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P 7e68e71d9d38456a9bf6d4e4887db304: Deleting consensus metadata
I20260808 02:25:13.281577 28346 catalog_manager.cc:4841] TS 7e68e71d9d38456a9bf6d4e4887db304 (127.19.231.3:41513): tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de (table test-table [id=6f9e1ff5374342f5804aa760f8fd33db]) successfully deleted
I20260808 02:25:13.283149 28959 ts_tablet_manager.cc:1939] T 3bc8d3e1722f45f8b3d7d2dbb98ad6de P c955d9ea567d48b4b04be23fa4800296: Deleting consensus metadata
I20260808 02:25:13.284863 28345 catalog_manager.cc:4841] TS c955d9ea567d48b4b04be23fa4800296 (127.19.231.2:38911): tablet 3bc8d3e1722f45f8b3d7d2dbb98ad6de (table test-table [id=6f9e1ff5374342f5804aa760f8fd33db]) successfully deleted
Aug 08 02:25:13 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.1@KRBTEST.COM
I20260808 02:25:13.303988 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.259111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.19.231.1:46063)
0808 02:25:13.259362 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:13.259371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:13.283381 (+ 24010us) server_negotiation.cc:407] Connection header received
0808 02:25:13.283615 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:13.283620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:13.283699 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:13.283823 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:13.286368 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.287449 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.289524 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.289963 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.290104 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.295627 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:13.295641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:13.295658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:13.295693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:13.300451 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:13.301197 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:13.301214 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:13.301217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:13.301478 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:13.301896 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:13.301901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:13.301905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:13.302342 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:13.302906 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:13.303110 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 02:25:13.303498 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 02:25:13.305164 28571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:13.456250 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 28433
I20260808 02:25:13.482792 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 28580
Aug 08 02:25:13 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.2@KRBTEST.COM
I20260808 02:25:13.505993 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.483829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.19.231.2:38911)
0808 02:25:13.484065 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:13.484074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:13.484101 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:25:13.484172 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:13.484178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:13.484281 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:13.484405 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:13.485738 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.487145 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.488539 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.489312 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.489446 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.498716 (+ 9270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:13.498737 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:13.498759 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:13.498806 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:13.502962 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:13.503739 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:13.503749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:13.503752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:13.503929 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:13.504346 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:13.504350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:13.504353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:13.504837 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:13.505171 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:13.505385 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 02:25:13.505644 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:25:13.506809 28715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:13.668514 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 28580
I20260808 02:25:13.700712 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 28724
Aug 08 02:25:13 dist-test-slave-pgkr krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.231.3@KRBTEST.COM
I20260808 02:25:13.717322 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:13.701493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.19.231.3:41513)
0808 02:25:13.701969 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:13.701974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:13.701990 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:25:13.702046 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:13.702050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:13.702117 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:13.702243 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:13.703779 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.704498 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.705856 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:13.706269 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:13.706366 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:13.711357 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:13.711373 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:13.711390 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:13.711427 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:13.714402 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:13.715098 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:13.715106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:13.715108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:13.715241 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:13.715662 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:13.715667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:13.715670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:13.716219 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:13.716492 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:13.716791 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 02:25:13.716999 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20260808 02:25:13.718036 28858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:13.851912 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 28724
I20260808 02:25:13.881078 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 28328
I20260808 02:25:13.882362 28401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:14.003580 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 28328
2026-08-08T02:25:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7503 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:14 dist-test-slave-pgkr krb5kdc[29012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:14 dist-test-slave-pgkr krb5kdc[29012](info): set up 2 sockets
Aug 08 02:25:14 dist-test-slave-pgkr krb5kdc[29012](info): commencing operation
krb5kdc: starting...
W20260808 02:25:15.168239 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:25:15 dist-test-slave-pgkr krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:25:15.451228 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41887
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:34941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:41887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:15.806999 29026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:15.807299 29026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:15.807390 29026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:15.814709 29026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:15.814826 29026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:15.814877 29026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:15.814942 29026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:15.814990 29026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:15.825960 29026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:34941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:41887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:15.827975 29026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:15.830045 29026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:15.843168 29032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:15.843300 29031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:15.843302 29035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:15.843871 29026 server_base.cc:1034] running on GCE node
I20260808 02:25:15.844669 29026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:15.846141 29026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:15.847343 29026 hybrid_clock.cc:648] HybridClock initialized: now 1786155915847302 us; error 25 us; skew 500 ppm
I20260808 02:25:15.847721 29026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T02:25:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1914 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:16 dist-test-slave-pgkr krb5kdc[29041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:16 dist-test-slave-pgkr krb5kdc[29041](info): set up 2 sockets
Aug 08 02:25:16 dist-test-slave-pgkr krb5kdc[29041](info): commencing operation
krb5kdc: starting...
W20260808 02:25:17.084864 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:25:17 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:25:17.373752 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45809
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:17.729275 29054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:17.729621 29054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:17.729727 29054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:17.737203 29054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:17.737329 29054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:17.737458 29054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:17.737579 29054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:17.737655 29054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:17.747696 29054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:45809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:17.749166 29054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:17.750594 29054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:17.763456 29059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:17.763561 29062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:17.763582 29060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:17.764734 29054 server_base.cc:1034] running on GCE node
I20260808 02:25:17.765534 29054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:17.767068 29054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:17.768301 29054 hybrid_clock.cc:648] HybridClock initialized: now 1786155917768249 us; error 39 us; skew 500 ppm
I20260808 02:25:17.768682 29054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:17 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155917, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:17.772197 29054 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:25:17.773873 29054 webserver.cc:460] Webserver started at http://127.19.231.62:39877/ using document root <none> and password file <none>
I20260808 02:25:17.774801 29054 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:17.774973 29054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:17.775362 29054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:17.778476 29054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "b848055a60e04639b19c49917604332f"
format_stamp: "Formatted at 2026-08-08 02:25:17 on dist-test-slave-pgkr"
server_key: "b7e680a7cb8ed51149145c4db87d2d56"
server_key_iv: "b5c2e40340f3ea931fbf1644a42ba193"
server_key_version: "encryptionkey@0"
I20260808 02:25:17.779349 29054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "b848055a60e04639b19c49917604332f"
format_stamp: "Formatted at 2026-08-08 02:25:17 on dist-test-slave-pgkr"
server_key: "b7e680a7cb8ed51149145c4db87d2d56"
server_key_iv: "b5c2e40340f3ea931fbf1644a42ba193"
server_key_version: "encryptionkey@0"
I20260808 02:25:17.784741 29054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260808 02:25:17.788218 29069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:17.789605 29054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:25:17.789793 29054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "b848055a60e04639b19c49917604332f"
format_stamp: "Formatted at 2026-08-08 02:25:17 on dist-test-slave-pgkr"
server_key: "b7e680a7cb8ed51149145c4db87d2d56"
server_key_iv: "b5c2e40340f3ea931fbf1644a42ba193"
server_key_version: "encryptionkey@0"
I20260808 02:25:17.789994 29054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:17.817137 29054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:17.821635 29054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:17.821959 29054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:17.841981 29054 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:45809
I20260808 02:25:17.842002 29122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:45809 every 8 connection(s)
I20260808 02:25:17.843444 29054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:17.847426 29123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:17.852367 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29054
I20260808 02:25:17.852641 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:25:17.853286 20380 external_mini_cluster.cc:1428] Setting key 9dccaa8de1a4ff3b633e76679257077c
I20260808 02:25:17.859925 29123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Bootstrap starting.
I20260808 02:25:17.863900 29123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:17.865034 29123 log.cc:824] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:17.868213 29123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: No bootstrap required, opened a new log
Aug 08 02:25:17 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:17.879653 29126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:17.855530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.19.231.62:45809)
0808 02:25:17.856505 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:17.856521 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:17.856579 (+ 58us) server_negotiation.cc:407] Connection header received
0808 02:25:17.859079 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:17.859109 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:17.859641 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:17.860094 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:17.861556 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:17.863607 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:17.864985 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:17.866010 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:17.866172 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:17.870388 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:17.870404 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:17.870430 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:17.870475 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:17.874884 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:17.875560 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:17.875568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:17.875578 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:17.875808 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:17.876211 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:17.876227 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:17.876231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:17.876998 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:17.877354 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:17.877663 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:25:17.878153 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":427,"threads_started":1}
I20260808 02:25:17.879434 29123 raft_consensus.cc:348] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:17.879997 29123 raft_consensus.cc:374] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:17.880147 29123 raft_consensus.cc:729] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b848055a60e04639b19c49917604332f, State: Initialized, Role: FOLLOWER
I20260808 02:25:17.880862 29123 consensus_queue.cc:260] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:17.881176 29123 raft_consensus.cc:388] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:17.881318 29123 raft_consensus.cc:482] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:17.881554 29123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:17.884471 29123 raft_consensus.cc:504] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:17.884943 29123 leader_election.cc:302] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b848055a60e04639b19c49917604332f; no voters:
I20260808 02:25:17.885485 29123 leader_election.cc:288] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:17.885643 29128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:17.886387 29128 raft_consensus.cc:686] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 LEADER]: Becoming Leader. State: Replica: b848055a60e04639b19c49917604332f, State: Running, Role: LEADER
I20260808 02:25:17.887003 29128 consensus_queue.cc:237] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:17.887727 29123 sys_catalog.cc:564] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:17.890136 29129 sys_catalog.cc:455] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b848055a60e04639b19c49917604332f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } } }
I20260808 02:25:17.890110 29130 sys_catalog.cc:455] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b848055a60e04639b19c49917604332f. Latest consensus state: current_term: 1 leader_uuid: "b848055a60e04639b19c49917604332f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } } }
I20260808 02:25:17.890597 29129 sys_catalog.cc:458] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:17.890646 29130 sys_catalog.cc:458] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:17.892140 29133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:17.897734 29133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:17.907260 29133 catalog_manager.cc:1347] Generated new cluster ID: 14aaf53e5597410593596a3e2833118d
I20260808 02:25:17.907377 29133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:17.922195 29133 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:17.924360 29133 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:17.946095 29133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Generated new TSK 0
I20260808 02:25:17.947257 29133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:25:18.013773 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45809
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:18.358472 29149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:18.358809 29149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:18.358991 29149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:18.366060 29149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:18.366158 29149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:18.366348 29149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:25:18.376073 29149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:18.377344 29149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:18.378773 29149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:18.392781 29155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:18.392746 29157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:18.392835 29154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:18.394008 29149 server_base.cc:1034] running on GCE node
I20260808 02:25:18.394726 29149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:18.395727 29149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:18.396925 29149 hybrid_clock.cc:648] HybridClock initialized: now 1786155918396868 us; error 47 us; skew 500 ppm
I20260808 02:25:18.397238 29149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:18 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155918, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:18.400538 29149 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:25:18.402009 29149 webserver.cc:460] Webserver started at http://127.19.231.1:43265/ using document root <none> and password file <none>
I20260808 02:25:18.402774 29149 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:18.402925 29149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:18.403245 29149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:18.406090 29149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "495df8bfefb5440cb8f5fa8b22d712d1"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "f6211a5ea3b117ca9162eaac8b0feec2"
server_key_iv: "a92be4ba2c12f7aec9717dea2cd7b693"
server_key_version: "encryptionkey@0"
I20260808 02:25:18.406944 29149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "495df8bfefb5440cb8f5fa8b22d712d1"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "f6211a5ea3b117ca9162eaac8b0feec2"
server_key_iv: "a92be4ba2c12f7aec9717dea2cd7b693"
server_key_version: "encryptionkey@0"
I20260808 02:25:18.412200 29149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 02:25:18.415606 29164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:18.416844 29149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:25:18.417017 29149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "495df8bfefb5440cb8f5fa8b22d712d1"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "f6211a5ea3b117ca9162eaac8b0feec2"
server_key_iv: "a92be4ba2c12f7aec9717dea2cd7b693"
server_key_version: "encryptionkey@0"
I20260808 02:25:18.417209 29149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:18.439482 29149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:18.444183 29149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:18.444499 29149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:18.445564 29149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:18.447026 29149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:18.447134 29149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:18.447294 29149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:18.447368 29149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:18.482280 29149 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:46799
I20260808 02:25:18.482340 29278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:46799 every 8 connection(s)
I20260808 02:25:18.483659 29149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:18.493548 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29149
I20260808 02:25:18.493780 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:25:18.494659 20380 external_mini_cluster.cc:1428] Setting key dc0b3074899b3de0bb48c086a125c4e8
Aug 08 02:25:18 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155918, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:18.508766 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:18.486364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:39917 (local address 127.19.231.62:45809)
0808 02:25:18.486871 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:18.486876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:18.487833 (+ 957us) server_negotiation.cc:407] Connection header received
0808 02:25:18.489880 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:18.489884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:18.489954 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:18.490054 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:18.491660 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:18.492410 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:18.494816 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:18.495261 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:18.495406 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:18.503077 (+ 7671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:18.503082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:18.503085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:18.503117 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:18.505640 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:18.506403 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:18.506407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:18.506409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:18.506534 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:18.507002 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:18.507006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:18.507008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:18.507265 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:18.507425 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:18.508161 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 02:25:18.508394 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":297,"threads_started":1}
I20260808 02:25:18.509753 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:18.486907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:39917)
0808 02:25:18.487623 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:18.487674 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:18.489596 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:18.490274 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:18.490282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:18.490910 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:18.491490 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:18.491512 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:18.492565 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:18.492571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:18.494636 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:18.494656 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:18.495498 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:18.495508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:18.495842 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:18.500528 (+ 4686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:18.500552 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:18.502833 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:18.505804 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:18.505813 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:18.505830 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:18.506264 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:18.506666 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:18.506671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:18.506675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:18.506871 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:18.507399 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:18.507407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:18.507760 (+ 353us) client_negotiation.cc:769] Sending connection context
0808 02:25:18.508032 (+ 272us) client_negotiation.cc:240] Negotiation successful
0808 02:25:18.508579 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":271,"threads_started":1}
I20260808 02:25:18.511411 29279 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45809
I20260808 02:25:18.511754 29279 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:18.512640 29279 heartbeater.cc:511] Master 127.19.231.62:45809 requested a full tablet report, sending...
I20260808 02:25:18.515249 29086 ts_manager.cc:194] Registered new tserver with Master: 495df8bfefb5440cb8f5fa8b22d712d1 (127.19.231.1:46799)
I20260808 02:25:18.517468 29086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:39917
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:25:18.559942 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45809
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:18.927345 29285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:18.927631 29285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:18.927728 29285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:18.935061 29285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:18.935173 29285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:18.935284 29285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:25:18.945103 29285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:18.946343 29285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:18.947827 29285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:18.961122 29291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:18.961122 29292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:18.961186 29294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:18.962183 29285 server_base.cc:1034] running on GCE node
I20260808 02:25:18.962895 29285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:18.963855 29285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:18.965087 29285 hybrid_clock.cc:648] HybridClock initialized: now 1786155918965025 us; error 58 us; skew 500 ppm
I20260808 02:25:18.965399 29285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:18 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155918, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:18.968761 29285 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:25:18.970325 29285 webserver.cc:460] Webserver started at http://127.19.231.2:34675/ using document root <none> and password file <none>
I20260808 02:25:18.971086 29285 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:18.971233 29285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:18.971561 29285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:18.974510 29285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "52a84cd21a7244c1a1090b2d0a2afce6"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "8cf35736a9e6abc97f07674b24298756"
server_key_iv: "c37a3b61779ed8e0a785bbed6a68721a"
server_key_version: "encryptionkey@0"
I20260808 02:25:18.975423 29285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "52a84cd21a7244c1a1090b2d0a2afce6"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "8cf35736a9e6abc97f07674b24298756"
server_key_iv: "c37a3b61779ed8e0a785bbed6a68721a"
server_key_version: "encryptionkey@0"
I20260808 02:25:18.980645 29285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 02:25:18.984107 29301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:18.985461 29285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:25:18.985682 29285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "52a84cd21a7244c1a1090b2d0a2afce6"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "8cf35736a9e6abc97f07674b24298756"
server_key_iv: "c37a3b61779ed8e0a785bbed6a68721a"
server_key_version: "encryptionkey@0"
I20260808 02:25:18.985913 29285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:19.002766 29285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:19.007272 29285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:19.007568 29285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:19.008572 29285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:19.010051 29285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:19.010154 29285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:19.010316 29285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:19.010390 29285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:19.043701 29285 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:38305
I20260808 02:25:19.043737 29420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:38305 every 8 connection(s)
I20260808 02:25:19.045066 29285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:19.049005 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29285
I20260808 02:25:19.049218 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:25:19.049765 20380 external_mini_cluster.cc:1428] Setting key a6d97d1c83cc81e3552d4d610e03ad7c
Aug 08 02:25:19 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155918, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:19.067112 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.047812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:46505 (local address 127.19.231.62:45809)
0808 02:25:19.048311 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:19.048315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:19.050700 (+ 2385us) server_negotiation.cc:407] Connection header received
0808 02:25:19.051289 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:19.051294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:19.051368 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:19.051487 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:19.053101 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.053837 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.055475 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.055909 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.056056 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:19.059802 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:19.059811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:19.059818 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:19.059864 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:19.063434 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.064438 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.064442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.064444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.064567 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.065051 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.065059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.065065 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.065455 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:19.065611 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:19.066315 (+ 704us) server_negotiation.cc:299] Negotiation successful
0808 02:25:19.066571 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260808 02:25:19.067840 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.048269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:46505)
0808 02:25:19.049021 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:19.049065 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:19.051029 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:19.051788 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:19.051796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:19.052313 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:19.052916 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:19.052933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.054016 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.054020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:19.055325 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:19.055337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.056154 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.056161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:19.056407 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:19.057104 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:19.057127 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:19.059558 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:19.063621 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:19.063633 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:19.063653 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:19.064312 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:19.064667 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:19.064671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:19.064674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:19.064906 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:19.065601 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:19.065607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:19.065922 (+ 315us) client_negotiation.cc:769] Sending connection context
0808 02:25:19.066148 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 02:25:19.066552 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":332,"threads_started":1}
I20260808 02:25:19.069463 29421 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45809
I20260808 02:25:19.069795 29421 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:19.070564 29421 heartbeater.cc:511] Master 127.19.231.62:45809 requested a full tablet report, sending...
I20260808 02:25:19.072376 29086 ts_manager.cc:194] Registered new tserver with Master: 52a84cd21a7244c1a1090b2d0a2afce6 (127.19.231.2:38305)
I20260808 02:25:19.073621 29086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:46505
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:25:19.114109 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45809
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:19.459594 29427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:19.459934 29427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:19.460062 29427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:19.466987 29427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:19.467085 29427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:19.467203 29427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:25:19.476624 29427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:19.477840 29427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:19.479126 29427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:19.492954 29432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:19.492992 29433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:19.493034 29435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:19.494079 29427 server_base.cc:1034] running on GCE node
I20260808 02:25:19.494819 29427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:19.495723 29427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:19.496948 29427 hybrid_clock.cc:648] HybridClock initialized: now 1786155919496889 us; error 53 us; skew 500 ppm
I20260808 02:25:19.497287 29427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:19 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155919, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:19.500590 29427 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:25:19.502130 29427 webserver.cc:460] Webserver started at http://127.19.231.3:46835/ using document root <none> and password file <none>
I20260808 02:25:19.502928 29427 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:19.503072 29427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:19.503400 29427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:19.506351 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "d1cdae69060f4b719fe3a24ce74e7452"
format_stamp: "Formatted at 2026-08-08 02:25:19 on dist-test-slave-pgkr"
server_key: "f4b380c4be49cccde2d25ba92df7f8ca"
server_key_iv: "d4a55ec70d63ef9883bb65b1c7e5641c"
server_key_version: "encryptionkey@0"
I20260808 02:25:19.507225 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "d1cdae69060f4b719fe3a24ce74e7452"
format_stamp: "Formatted at 2026-08-08 02:25:19 on dist-test-slave-pgkr"
server_key: "f4b380c4be49cccde2d25ba92df7f8ca"
server_key_iv: "d4a55ec70d63ef9883bb65b1c7e5641c"
server_key_version: "encryptionkey@0"
I20260808 02:25:19.512851 29427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:25:19.516381 29443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:19.517752 29427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:25:19.517928 29427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "d1cdae69060f4b719fe3a24ce74e7452"
format_stamp: "Formatted at 2026-08-08 02:25:19 on dist-test-slave-pgkr"
server_key: "f4b380c4be49cccde2d25ba92df7f8ca"
server_key_iv: "d4a55ec70d63ef9883bb65b1c7e5641c"
server_key_version: "encryptionkey@0"
I20260808 02:25:19.518146 29427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:19.521481 29279 heartbeater.cc:503] Master 127.19.231.62:45809 was elected leader, sending a full tablet report...
I20260808 02:25:19.533391 29427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:19.537524 29427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:19.537829 29427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:19.538889 29427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:19.540270 29427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:19.540383 29427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:19.540535 29427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:19.540616 29427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:19.575978 29427 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:41469
I20260808 02:25:19.575997 29558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:41469 every 8 connection(s)
I20260808 02:25:19.577342 29427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:19.582387 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29427
I20260808 02:25:19.582577 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:25:19.583077 20380 external_mini_cluster.cc:1428] Setting key de99aaee9463e6e7c8f8718307ddd2e0
Aug 08 02:25:19 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155919, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:19.598791 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.579942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:50095 (local address 127.19.231.62:45809)
0808 02:25:19.580395 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:19.580400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:19.581379 (+ 979us) server_negotiation.cc:407] Connection header received
0808 02:25:19.583547 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:19.583553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:19.583662 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:19.583784 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:19.585548 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.586455 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.588324 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.588723 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.588819 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:19.592563 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:19.592567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:19.592571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:19.592601 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:19.595557 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.596218 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.596222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.596224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.596346 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.596864 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.596871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.596877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.597168 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:19.597309 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:19.598109 (+ 800us) server_negotiation.cc:299] Negotiation successful
0808 02:25:19.598296 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260808 02:25:19.599761 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.580494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:50095)
0808 02:25:19.581213 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:19.581257 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:19.583281 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:19.584075 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:19.584086 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:19.584653 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:19.585366 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:19.585391 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.586703 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.586707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:19.588202 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:19.588215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.588964 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.588972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:19.589266 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:19.590026 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:19.590083 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:19.592373 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:19.595689 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:19.595695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:19.595707 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:19.596114 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:19.596454 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:19.596459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:19.596462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:19.596713 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:19.597303 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:19.597312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:19.597696 (+ 384us) client_negotiation.cc:769] Sending connection context
0808 02:25:19.597981 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 02:25:19.598527 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":281,"threads_started":1}
I20260808 02:25:19.601148 29559 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45809
I20260808 02:25:19.601456 29559 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:19.602181 29559 heartbeater.cc:511] Master 127.19.231.62:45809 requested a full tablet report, sending...
I20260808 02:25:19.603659 29086 ts_manager.cc:194] Registered new tserver with Master: d1cdae69060f4b719fe3a24ce74e7452 (127.19.231.3:41469)
I20260808 02:25:19.604414 29086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:50095
I20260808 02:25:19.611243 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab.
Entry for principal oryx/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab.
WARNING: no policy specified for oryx/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab.
Entry for principal oryx/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab.
WARNING: no policy specified for oryx/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab.
Entry for principal oryx/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab.
I20260808 02:25:19.786115 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29149
Aug 08 02:25:19 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:25:19.804399 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.787565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.19.231.1:46799)
0808 02:25:19.788678 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:19.788686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:19.788714 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:25:19.788779 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:19.788786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:19.789010 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:19.789177 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:19.790771 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.791811 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.793098 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.793761 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.793859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:19.798010 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:19.798021 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:19.798035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:19.798069 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:19.801029 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.801728 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.801735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.801738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.801867 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.802270 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.802273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.802276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.802845 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:19.803226 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:19.803856 (+ 630us) server_negotiation.cc:299] Negotiation successful
0808 02:25:19.804069 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":728,"threads_started":1}
I20260808 02:25:19.805716 29274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:19.899904 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29149
I20260808 02:25:19.921926 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29285
Aug 08 02:25:19 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:25:19.938726 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:19.923027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.19.231.2:38305)
0808 02:25:19.923654 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:19.923662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:19.923687 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:25:19.923752 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:19.923759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:19.923977 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:19.924155 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:19.925466 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.926521 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.927793 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:19.928355 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:19.928496 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:19.932832 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:19.932844 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:19.932858 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:19.932891 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:19.935769 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.936450 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.936458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.936460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.936582 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:19.937010 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:19.937015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:19.937017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:19.937364 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:19.937720 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:19.938012 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 02:25:19.938297 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":361,"threads_started":1}
I20260808 02:25:19.940111 29416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:20.022945 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29285
I20260808 02:25:20.043556 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29427
Aug 08 02:25:20 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:25:20.060479 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:20.044572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.19.231.3:41469)
0808 02:25:20.045124 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:20.045132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:20.045177 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:25:20.045240 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:20.045246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:20.045471 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:20.045603 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:20.047062 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:20.048068 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:20.049412 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:20.049983 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:20.050083 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:20.054841 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:20.054855 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:20.054871 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:20.054907 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:20.057678 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:20.058308 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:20.058314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:20.058317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:20.058432 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:20.058898 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:20.058901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:20.058903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:20.059258 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:20.059482 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:20.059844 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 02:25:20.060049 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
I20260808 02:25:20.061826 29554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:20.144515 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29427
I20260808 02:25:20.164913 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29054
I20260808 02:25:20.166144 29118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:20.271764 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29054
I20260808 02:25:20.297562 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45809
--webserver_interface=127.19.231.62
--webserver_port=39877
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:20.641088 29577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:20.641376 29577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:20.641492 29577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:20.648896 29577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:20.649017 29577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:20.649137 29577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:20.649245 29577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:20.649322 29577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:20.659847 29577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:45809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=39877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:20.661257 29577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:20.662691 29577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:20.675308 29582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:20.675242 29583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:20.675272 29585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:20.676153 29577 server_base.cc:1034] running on GCE node
I20260808 02:25:20.676944 29577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:20.678361 29577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:20.679601 29577 hybrid_clock.cc:648] HybridClock initialized: now 1786155920679513 us; error 69 us; skew 500 ppm
I20260808 02:25:20.679992 29577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:20 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155920, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:20.683146 29577 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:25:20.684861 29577 webserver.cc:460] Webserver started at http://127.19.231.62:39877/ using document root <none> and password file <none>
I20260808 02:25:20.685608 29577 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:20.685808 29577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:20.691298 29577 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:25:20.694128 29592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:20.695575 29577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:25:20.695785 29577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "b848055a60e04639b19c49917604332f"
format_stamp: "Formatted at 2026-08-08 02:25:17 on dist-test-slave-pgkr"
server_key: "b7e680a7cb8ed51149145c4db87d2d56"
server_key_iv: "b5c2e40340f3ea931fbf1644a42ba193"
server_key_version: "encryptionkey@0"
I20260808 02:25:20.696797 29577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:20.708068 29577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:20.712522 29577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:20.712798 29577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:20.731839 29577 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:45809
I20260808 02:25:20.731848 29645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:45809 every 8 connection(s)
I20260808 02:25:20.733403 29577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:20.734810 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29577
I20260808 02:25:20.735894 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:46799
--local_ip_for_outbound_sockets=127.19.231.1
--tserver_master_addrs=127.19.231.62:45809
--webserver_port=43265
--webserver_interface=127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:25:20.743448 29647 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:25:20.748219 29647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Bootstrap starting.
I20260808 02:25:20.773422 29647 log.cc:824] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:20.782387 29647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:25:20.783128 29647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Bootstrap complete.
I20260808 02:25:20.793010 29647 raft_consensus.cc:348] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:20.793964 29647 raft_consensus.cc:729] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b848055a60e04639b19c49917604332f, State: Initialized, Role: FOLLOWER
I20260808 02:25:20.794754 29647 consensus_queue.cc:260] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:20.795038 29647 raft_consensus.cc:388] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:20.795188 29647 raft_consensus.cc:482] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:20.795396 29647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:25:20.799566 29647 raft_consensus.cc:504] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:20.800179 29647 leader_election.cc:302] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b848055a60e04639b19c49917604332f; no voters:
I20260808 02:25:20.800936 29647 leader_election.cc:288] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:25:20.801139 29651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:25:20.802573 29651 raft_consensus.cc:686] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [term 2 LEADER]: Becoming Leader. State: Replica: b848055a60e04639b19c49917604332f, State: Running, Role: LEADER
I20260808 02:25:20.803261 29651 consensus_queue.cc:237] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } }
I20260808 02:25:20.803606 29647 sys_catalog.cc:564] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:20.807304 29653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b848055a60e04639b19c49917604332f. Latest consensus state: current_term: 2 leader_uuid: "b848055a60e04639b19c49917604332f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } } }
I20260808 02:25:20.807662 29653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:20.808216 29652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b848055a60e04639b19c49917604332f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b848055a60e04639b19c49917604332f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45809 } } }
I20260808 02:25:20.808516 29652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:20.816136 29656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:20.825172 29656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:20.827271 29656 catalog_manager.cc:1259] Loaded cluster ID: 14aaf53e5597410593596a3e2833118d
I20260808 02:25:20.827400 29656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:20.833309 29656 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:20.835723 29656 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b848055a60e04639b19c49917604332f: Loaded TSK: 0
I20260808 02:25:20.841248 29656 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:25:21.138226 29649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:21.138513 29649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:21.138693 29649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:21.146118 29649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:21.146229 29649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:21.146399 29649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:25:21.156527 29649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.1.keytab
--principal=oryx/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:46799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=43265
--tserver_master_addrs=127.19.231.62:45809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:21.157928 29649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:21.159309 29649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:21.174696 29674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:21.174695 29673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:21.176035 29649 server_base.cc:1034] running on GCE node
W20260808 02:25:21.178880 29676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:21.179641 29649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:21.180791 29649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:21.182024 29649 hybrid_clock.cc:648] HybridClock initialized: now 1786155921181955 us; error 64 us; skew 500 ppm
I20260808 02:25:21.182363 29649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:21 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155921, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:21.185755 29649 init.cc:377] Logged in from keytab as oryx/127.19.231.1@KRBTEST.COM (short username oryx)
I20260808 02:25:21.187480 29649 webserver.cc:460] Webserver started at http://127.19.231.1:43265/ using document root <none> and password file <none>
I20260808 02:25:21.188192 29649 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:21.188338 29649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:21.194008 29649 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:25:21.197274 29683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:21.198781 29649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:25:21.198956 29649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "495df8bfefb5440cb8f5fa8b22d712d1"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "f6211a5ea3b117ca9162eaac8b0feec2"
server_key_iv: "a92be4ba2c12f7aec9717dea2cd7b693"
server_key_version: "encryptionkey@0"
I20260808 02:25:21.199893 29649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:21.224167 29649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:21.228696 29649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:21.228989 29649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:21.230017 29649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:21.231469 29649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:21.231595 29649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:21.231719 29649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:21.231817 29649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:21.277714 29649 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:46799
I20260808 02:25:21.277822 29799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:46799 every 8 connection(s)
I20260808 02:25:21.279328 29649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:21.286495 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29649
I20260808 02:25:21.287204 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:38305
--local_ip_for_outbound_sockets=127.19.231.2
--tserver_master_addrs=127.19.231.62:45809
--webserver_port=34675
--webserver_interface=127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:25:21 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:21 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:25:21.682770 29804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:21.683028 29804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:21.683185 29804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:21.690316 29804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:21.690415 29804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:21.690667 29804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:25:21.700920 29804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.2.keytab
--principal=oryx/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:38305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=34675
--tserver_master_addrs=127.19.231.62:45809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:21.702237 29804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:21.703837 29804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:21.722520 29811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:21.722677 29810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:21.723100 29813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:21.723412 29804 server_base.cc:1034] running on GCE node
I20260808 02:25:21.724102 29804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:21.725093 29804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:21.726440 29804 hybrid_clock.cc:648] HybridClock initialized: now 1786155921726324 us; error 84 us; skew 500 ppm
I20260808 02:25:21.726825 29804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:21 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155921, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:21.730044 29804 init.cc:377] Logged in from keytab as oryx/127.19.231.2@KRBTEST.COM (short username oryx)
I20260808 02:25:21.731814 29804 webserver.cc:460] Webserver started at http://127.19.231.2:34675/ using document root <none> and password file <none>
I20260808 02:25:21.732522 29804 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:21.732686 29804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:21.738101 29804 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 02:25:21.740885 29820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:21.742285 29804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:25:21.742470 29804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "52a84cd21a7244c1a1090b2d0a2afce6"
format_stamp: "Formatted at 2026-08-08 02:25:18 on dist-test-slave-pgkr"
server_key: "8cf35736a9e6abc97f07674b24298756"
server_key_iv: "c37a3b61779ed8e0a785bbed6a68721a"
server_key_version: "encryptionkey@0"
I20260808 02:25:21.743495 29804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:21.757484 29804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:21.761792 29804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:21.762075 29804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:21.763192 29804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:21.764591 29804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:21.764721 29804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:21.764842 29804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:21.764932 29804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:21.807312 29804 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:38305
I20260808 02:25:21.807482 29935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:38305 every 8 connection(s)
I20260808 02:25:21.809026 29804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:21.818884 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29804
I20260808 02:25:21.820001 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:41469
--local_ip_for_outbound_sockets=127.19.231.3
--tserver_master_addrs=127.19.231.62:45809
--webserver_port=46835
--webserver_interface=127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:25:21 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:21 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:25:22.516407 29942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:22.516888 29942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:22.517096 29942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:22.531853 29942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:22.531999 29942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:22.532212 29942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:25:22.543203 29942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/krb5kdc/oryx_127.19.231.3.keytab
--principal=oryx/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:41469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=46835
--tserver_master_addrs=127.19.231.62:45809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:22.544726 29942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:22.546095 29942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:22.565557 29947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:22.571373 29948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:22.572855 29950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:22.573906 29942 server_base.cc:1034] running on GCE node
I20260808 02:25:22.574743 29942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:22.575891 29942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:22.577207 29942 hybrid_clock.cc:648] HybridClock initialized: now 1786155922577124 us; error 65 us; skew 500 ppm
I20260808 02:25:22.577706 29942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155922, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:22.581308 29942 init.cc:377] Logged in from keytab as oryx/127.19.231.3@KRBTEST.COM (short username oryx)
I20260808 02:25:22.584146 29942 webserver.cc:460] Webserver started at http://127.19.231.3:46835/ using document root <none> and password file <none>
I20260808 02:25:22.585048 29942 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:22.585225 29942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:22.592954 29942 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:25:22.597030 29958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.599090 29942 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 02:25:22.599260 29942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "d1cdae69060f4b719fe3a24ce74e7452"
format_stamp: "Formatted at 2026-08-08 02:25:19 on dist-test-slave-pgkr"
server_key: "f4b380c4be49cccde2d25ba92df7f8ca"
server_key_iv: "d4a55ec70d63ef9883bb65b1c7e5641c"
server_key_version: "encryptionkey@0"
I20260808 02:25:22.601135 29942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:22.621572 29942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:22.626220 29942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:22.626570 29942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:22.627784 29942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:22.629374 29942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:22.629494 29942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.629643 29942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:22.629740 29942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:22.683082 29942 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:41469
I20260808 02:25:22.683257 30074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:41469 every 8 connection(s)
I20260808 02:25:22.685091 29942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:22.693001 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 29942
W20260808 02:25:22.698107 29803 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.405s user 0.002s sys 0.013s
I20260808 02:25:22.700121 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:21.282511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:36875)
0808 02:25:21.283372 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:21.283422 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:21.285199 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:21.286708 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:21.286720 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:21.287397 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:21.287981 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:21.287999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.289607 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.289617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:21.290908 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:21.290921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.291978 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.291986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:21.292291 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:21.293001 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:21.293025 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.699197 (+1406172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":337,"threads_started":1}
W20260808 02:25:22.701247 29800 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:45809 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.231.62:45809: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
W20260808 02:25:22.702165 29802 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:21.282411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:36875 (local address 127.19.231.62:45809)
0808 02:25:21.283305 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:21.283325 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:21.283547 (+ 222us) server_negotiation.cc:407] Connection header received
0808 02:25:21.285767 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:21.285780 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:21.286167 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:21.286539 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:21.288194 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.289438 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.291065 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.291732 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.291936 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.701229 (+1409293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:36875: BlockingRecv error: recv got EOF from 127.19.231.1:36875 (error 108)
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":316,"threads_started":1}
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:22.725872 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.702070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:49065)
0808 02:25:22.702647 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.702669 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.702822 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.713803 (+ 10981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.713810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.714357 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.714675 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.714693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.716460 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.716467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.718344 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.718365 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.719447 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.719455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.719681 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.720547 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.720578 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.725419 (+ 4841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409}
W20260808 02:25:22.726593 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:22.707736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:49065 (local address 127.19.231.62:45809)
0808 02:25:22.713267 (+ 5531us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.713278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.713306 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:25:22.713395 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.713405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.713529 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.714897 (+ 1368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.715235 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.716275 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.718557 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.719181 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.722063 (+ 2882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.726158 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:49065: BlockingRecv error: recv got EOF from 127.19.231.1:49065 (error 108)
Metrics: {"server-negotiator.queue_time_us":5303}
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:22.750581 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.695040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.19.231.62:45809)
0808 02:25:22.707836 (+ 12796us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.707847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.708611 (+ 764us) server_negotiation.cc:407] Connection header received
0808 02:25:22.708719 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.708729 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.708856 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.709008 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.717091 (+ 8083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.718503 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.721939 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.722556 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.722763 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.731029 (+ 8266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:22.731045 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:22.731076 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:22.731147 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:22.743868 (+ 12721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.745298 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.745330 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.745342 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.745653 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:22.747347 (+ 1694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:22.747352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:22.747356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:22.748808 (+ 1452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:22.749160 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:22.749718 (+ 558us) server_negotiation.cc:299] Negotiation successful
0808 02:25:22.750131 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12613,"thread_start_us":12461,"threads_started":1}
I20260808 02:25:22.751077 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.727112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:57889)
0808 02:25:22.731953 (+ 4841us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.731976 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.732116 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.734079 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.734088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.737963 (+ 3875us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.738413 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.738433 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.740945 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.740952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.742826 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.742847 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.744326 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.744333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.744707 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.745453 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.745479 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.750756 (+ 5277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4694,"thread_start_us":4472,"threads_started":1}
W20260808 02:25:22.751837 29800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:25:22.752094 30078 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:22.727923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:57889 (local address 127.19.231.62:45809)
0808 02:25:22.731595 (+ 3672us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.731602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.732223 (+ 621us) server_negotiation.cc:407] Connection header received
0808 02:25:22.732291 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.732297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.732367 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.732458 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.738698 (+ 6240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.740718 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.743045 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.743837 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.744027 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.751835 (+ 7808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:57889: BlockingRecv error: recv got EOF from 127.19.231.1:57889 (error 108)
Metrics: {"server-negotiator.queue_time_us":3533,"thread_start_us":9608,"threads_started":1}
W20260808 02:25:22.972782 29940 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.150s user 0.009s sys 0.005s
W20260808 02:25:22.974361 29939 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:21.812830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:33351 (local address 127.19.231.62:45809)
0808 02:25:21.813532 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:21.813543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:21.814474 (+ 931us) server_negotiation.cc:407] Connection header received
0808 02:25:21.816766 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:21.816775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:21.816918 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:21.817117 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:21.818695 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.819568 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.821235 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.821640 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.821768 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.973970 (+1152202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:33351: BlockingRecv error: recv got EOF from 127.19.231.2:33351 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":308,"threads_started":1}
I20260808 02:25:22.975040 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:21.813350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:33351)
0808 02:25:21.814294 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:21.814343 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:21.816460 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:21.817322 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:21.817331 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:21.817861 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:21.818413 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:21.818430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.819726 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.819732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:21.821087 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:21.821105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:21.821872 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:21.821882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:21.822113 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:21.822835 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:21.822857 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.973676 (+1150819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":385,"threads_started":1}
W20260808 02:25:22.976296 29937 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:45809 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.231.62:45809: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:22.988061 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.977188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:45169)
0808 02:25:22.977512 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.977528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.977679 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.978054 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.978059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.978483 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.978725 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.978737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.979875 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.979882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.980950 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.980962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.981797 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.981804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.981992 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.982807 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.982826 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.987599 (+ 4773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
W20260808 02:25:22.988166 29939 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:22.977196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:45169 (local address 127.19.231.62:45809)
0808 02:25:22.977385 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.977393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.977612 (+ 219us) server_negotiation.cc:407] Connection header received
0808 02:25:22.977835 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.977839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.977904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.978015 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.978874 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.979717 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.981112 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.981595 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.981737 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.987888 (+ 6151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:45169: BlockingRecv error: recv got EOF from 127.19.231.2:45169 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:22 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 02:25:22.999809 29939 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:22.989702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:34113 (local address 127.19.231.62:45809)
0808 02:25:22.989911 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.989920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.990005 (+ 85us) server_negotiation.cc:407] Connection header received
0808 02:25:22.990214 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.990219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.990320 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.990446 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.991233 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.992042 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.993452 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.993907 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.994052 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.999586 (+ 5534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:34113: BlockingRecv error: recv got EOF from 127.19.231.2:34113 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 02:25:22.999837 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.989201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:34113)
0808 02:25:22.989899 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.989918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.990077 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.990482 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.990487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.990910 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.991081 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.991091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.992208 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.992214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.993288 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.993300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.994080 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.994086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.994291 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.995046 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.995066 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:22.999408 (+ 4342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570}
W20260808 02:25:23.000324 29937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 02:25:23 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:23 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:23.770689 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.752533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:46951)
0808 02:25:23.753148 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:23.753173 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:23.753321 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:23.753821 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:23.753826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:23.754401 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:23.754664 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.754681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.756213 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.756223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.758791 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.758818 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.759772 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.759782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.760097 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.764488 (+ 4391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:23.764531 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:23.770235 (+ 5704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":346,"threads_started":1}
W20260808 02:25:23.771039 30085 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:23.752655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:46951 (local address 127.19.231.62:45809)
0808 02:25:23.753494 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:23.753499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:23.753514 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:25:23.753565 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:23.753569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:23.753639 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:23.753765 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:23.754811 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.755842 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.759094 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.759554 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.759671 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.770458 (+ 10787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:46951: BlockingRecv error: recv got EOF from 127.19.231.1:46951 (error 108)
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":517,"threads_started":1}
W20260808 02:25:23.924677 30077 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.190s user 0.000s sys 0.028s
W20260808 02:25:23.925925 29802 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:22.690767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:41907 (local address 127.19.231.62:45809)
0808 02:25:22.702542 (+ 11775us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:22.702548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:22.714636 (+ 12088us) server_negotiation.cc:407] Connection header received
0808 02:25:22.719106 (+ 4470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:22.719112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:22.719195 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:22.719302 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:22.723874 (+ 4572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.725584 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.731701 (+ 6117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.732378 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.732462 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.925686 (+1193224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:41907: BlockingRecv error: recv got EOF from 127.19.231.3:41907 (error 108)
Metrics: {"server-negotiator.queue_time_us":11662,"thread_start_us":22065,"threads_started":1}
I20260808 02:25:23.926529 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:22.689929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:41907)
0808 02:25:22.714348 (+ 24419us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:22.714446 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:22.718687 (+ 4241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:22.719833 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:22.719848 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:22.720846 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:22.723569 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.723627 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.727084 (+ 3457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.727093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.730721 (+ 3628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:22.730746 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:22.732911 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:22.732922 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:22.733411 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:22.734868 (+ 1457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:22.734923 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:23.925479 (+1190556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23923,"thread_start_us":23608,"threads_started":1}
W20260808 02:25:23.927661 30075 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:45809 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.231.62:45809: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Aug 08 02:25:23 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:23 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:23.937644 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.928449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:59365)
0808 02:25:23.928756 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:23.928779 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:23.928925 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:23.929285 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:23.929289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:23.929695 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:23.929875 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.929884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.931001 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.931005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.932099 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.932112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.932854 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.932859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.932985 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.933678 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:23.933697 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:23.937407 (+ 3710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176}
W20260808 02:25:23.937878 29802 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:23.928744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:59365 (local address 127.19.231.62:45809)
0808 02:25:23.928969 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:23.928974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:23.928989 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:25:23.929041 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:23.929045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:23.929108 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:23.929188 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:23.930017 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.930866 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.932236 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.932662 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.932765 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.937613 (+ 4848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:59365: BlockingRecv error: recv got EOF from 127.19.231.3:59365 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 08 02:25:23 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:23 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:23.946910 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:23.938412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:32937)
0808 02:25:23.938586 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:23.938605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:23.938730 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:23.939070 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:23.939074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:23.939489 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:23.939724 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.939735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.941037 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.941041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.942006 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:23.942017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.942769 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.942773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:23.942894 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.943410 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:23.943427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:23.946710 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:25:23.947094 29802 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:23.938498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:32937 (local address 127.19.231.62:45809)
0808 02:25:23.938753 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:23.938761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:23.938784 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:25:23.938837 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:23.938842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:23.938909 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:23.939022 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:23.939873 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.940898 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.942147 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:23.942582 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:23.942760 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:23.946848 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:32937: BlockingRecv error: recv got EOF from 127.19.231.3:32937 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 02:25:23.947256 30075 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 02:25:24 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:24 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:24.010025 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:24.000870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:49867)
0808 02:25:24.001480 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:24.001497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:24.001606 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:24.001902 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:24.001905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:24.002342 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:24.002488 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:24.002496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.003635 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.003639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:24.004731 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:24.004745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.005466 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.005470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:24.005588 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:24.006233 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:24.006250 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:24.009782 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":352,"threads_started":1}
W20260808 02:25:24.010268 29802 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:24.000977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:49867 (local address 127.19.231.62:45809)
0808 02:25:24.001176 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:24.001180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:24.001564 (+ 384us) server_negotiation.cc:407] Connection header received
0808 02:25:24.001710 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:24.001714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:24.001777 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:24.001858 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:24.002655 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.003511 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.004897 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.005313 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.005407 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:24.009990 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:49867: BlockingRecv error: recv got EOF from 127.19.231.2:49867 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 02:25:24 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:24 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:24.781368 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:24.771623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:40695)
0808 02:25:24.772177 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:24.772193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:24.772309 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:24.772625 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:24.772628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:24.773009 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:24.773201 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:24.773211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.774306 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.774311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:24.775573 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:24.775588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.776263 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.776267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:24.776398 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:24.777128 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:24.777148 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:24.781116 (+ 3968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
W20260808 02:25:24.781565 30088 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:24.771722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:40695 (local address 127.19.231.62:45809)
0808 02:25:24.772231 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:24.772241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:24.772269 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:25:24.772413 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:24.772416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:24.772488 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:24.772571 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:24.773348 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.774143 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.775712 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.776123 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.776218 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:24.781313 (+ 5095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:40695: BlockingRecv error: recv got EOF from 127.19.231.1:40695 (error 108)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":282,"threads_started":1}
Aug 08 02:25:24 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:24 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:24.957127 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:24.947847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:57191)
0808 02:25:24.948476 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:24.948492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:24.948654 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:24.948970 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:24.948974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:24.949412 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:24.949609 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:24.949618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.950600 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.950604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:24.951740 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:24.951753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.952490 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.952495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:24.952634 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:24.953221 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:24.953240 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:24.956740 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":387,"threads_started":1}
W20260808 02:25:24.957235 30088 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:24.947934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:57191 (local address 127.19.231.62:45809)
0808 02:25:24.948111 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:24.948117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:24.948592 (+ 475us) server_negotiation.cc:407] Connection header received
0808 02:25:24.948785 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:24.948789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:24.948850 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:24.948926 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:24.949735 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.950466 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.951866 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:24.952283 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:24.952375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:24.956960 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:57191: BlockingRecv error: recv got EOF from 127.19.231.3:57191 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 02:25:25 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:25 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:25.020550 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:25.010940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:46363)
0808 02:25:25.011428 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:25.011445 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:25.011612 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:25.011915 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:25.011918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:25.012345 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:25.012528 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:25.012537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.013784 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.013789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:25.014805 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:25.014818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.015474 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.015479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:25.015651 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:25.016236 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:25.016255 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:25.020193 (+ 3938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":285,"threads_started":1}
W20260808 02:25:25.020641 30088 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:25.011055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:46363 (local address 127.19.231.62:45809)
0808 02:25:25.011310 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:25.011317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:25.011528 (+ 211us) server_negotiation.cc:407] Connection header received
0808 02:25:25.011722 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:25.011725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:25.011789 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:25.011867 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:25.012710 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.013444 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.014945 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.015324 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.015416 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:25.020373 (+ 4957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:46363: BlockingRecv error: recv got EOF from 127.19.231.2:46363 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 08 02:25:25 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:25 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:25.791555 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:25.782280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:41737)
0808 02:25:25.782865 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:25.782888 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:25.783020 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:25.783394 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:25.783399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:25.783822 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:25.783989 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:25.783999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.785024 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.785036 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:25.785951 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:25.785962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.786654 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.786660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:25.786863 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:25.787497 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:25.787515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:25.791224 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
W20260808 02:25:25.791682 30092 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:25.782498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:41737 (local address 127.19.231.62:45809)
0808 02:25:25.783000 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:25.783005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:25.783023 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:25:25.783171 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:25.783177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:25.783250 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:25.783362 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:25.784139 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.784877 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.786093 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.786485 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.786616 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:25.791408 (+ 4792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:41737: BlockingRecv error: recv got EOF from 127.19.231.1:41737 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
Aug 08 02:25:25 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:25 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:25.968047 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:25.958112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:43987)
0808 02:25:25.958745 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:25.958764 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:25.958909 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:25.959216 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:25.959219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:25.959768 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:25.959988 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:25.959999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.960984 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.960988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:25.962027 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:25.962041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.962729 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.962736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:25.962957 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:25.963592 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:25.963613 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:25.967689 (+ 4076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":328,"threads_started":1}
W20260808 02:25:25.968091 30092 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:25.958177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:43987 (local address 127.19.231.62:45809)
0808 02:25:25.958349 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:25.958353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:25.958841 (+ 488us) server_negotiation.cc:407] Connection header received
0808 02:25:25.959009 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:25.959012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:25.959073 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:25.959153 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:25.960126 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.960855 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.962158 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:25.962534 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:25.962687 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:25.967909 (+ 5222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:43987: BlockingRecv error: recv got EOF from 127.19.231.3:43987 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 02:25:26 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:26 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:26.030823 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:26.021596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:42781)
0808 02:25:26.022104 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:26.022121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:26.022245 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:26.022598 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:26.022602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:26.022997 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:26.023172 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:26.023182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.024412 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.024416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:26.025334 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:26.025346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.026195 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.026204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:26.026364 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:26.027083 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:26.027100 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:26.030535 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
W20260808 02:25:26.031042 30092 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:26.021700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:42781 (local address 127.19.231.62:45809)
0808 02:25:26.021944 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:26.021951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:26.022207 (+ 256us) server_negotiation.cc:407] Connection header received
0808 02:25:26.022344 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:26.022347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:26.022423 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:26.022508 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:26.023333 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.024260 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.025456 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.026022 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.026137 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:26.030798 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:42781: BlockingRecv error: recv got EOF from 127.19.231.2:42781 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 02:25:26 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:26 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:26.801889 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:26.792540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:49597)
0808 02:25:26.793087 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:26.793105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:26.793210 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:26.793520 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:26.793524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:26.793938 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:26.794098 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:26.794107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.795234 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.795237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:26.796242 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:26.796254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.796986 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.796990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:26.797119 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:26.797759 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:26.797776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:26.801621 (+ 3845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":295,"threads_started":1}
W20260808 02:25:26.802114 30096 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:26.792616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:49597 (local address 127.19.231.62:45809)
0808 02:25:26.793223 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:26.793227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:26.793267 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:25:26.793313 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:26.793316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:26.793375 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:26.793468 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:26.794246 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.795099 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.796386 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.796860 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.796953 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:26.801883 (+ 4930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:49597: BlockingRecv error: recv got EOF from 127.19.231.1:49597 (error 108)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":346,"threads_started":1}
Aug 08 02:25:26 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:26 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:26.977994 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:26.969114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:35377)
0808 02:25:26.969764 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:26.969790 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:26.969922 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:26.970255 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:26.970259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:26.970655 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:26.970856 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:26.970865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.971832 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.971836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:26.972753 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:26.972764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.973392 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.973397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:26.973532 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:26.974144 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:26.974162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:26.977747 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":358,"threads_started":1}
W20260808 02:25:26.978207 30096 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:26.969137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:35377 (local address 127.19.231.62:45809)
0808 02:25:26.969357 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:26.969365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:26.969885 (+ 520us) server_negotiation.cc:407] Connection header received
0808 02:25:26.970027 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:26.970030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:26.970093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:26.970190 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:26.970985 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.971708 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.972869 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:26.973269 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:26.973357 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:26.977938 (+ 4581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:35377: BlockingRecv error: recv got EOF from 127.19.231.3:35377 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 08 02:25:27 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:27 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:27.041100 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:27.031870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:38611)
0808 02:25:27.032441 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:27.032457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:27.032609 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:27.032971 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:27.032976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:27.033391 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:27.033562 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:27.033571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.034675 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.034681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:27.035678 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:27.035691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.036412 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.036418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:27.036596 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:27.037189 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:27.037206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:27.040727 (+ 3521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":318,"threads_started":1}
W20260808 02:25:27.041227 30096 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:27.031951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:38611 (local address 127.19.231.62:45809)
0808 02:25:27.032194 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:27.032203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:27.032540 (+ 337us) server_negotiation.cc:407] Connection header received
0808 02:25:27.032740 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:27.032752 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:27.032828 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:27.032934 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:27.033713 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.034505 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.035828 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.036263 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.036400 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:27.040931 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:38611: BlockingRecv error: recv got EOF from 127.19.231.2:38611 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 02:25:27 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:27 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:27.812805 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:27.802843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:35957)
0808 02:25:27.803865 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:27.803881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:27.803998 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:27.804316 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:27.804319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:27.804823 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:27.805022 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:27.805032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.806081 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.806085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:27.807145 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:27.807170 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.808223 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.808227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:27.808359 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:27.809022 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:27.809040 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:27.812578 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":357,"threads_started":1}
W20260808 02:25:27.813278 30099 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:27.802847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:35957 (local address 127.19.231.62:45809)
0808 02:25:27.803695 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:27.803700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:27.803957 (+ 257us) server_negotiation.cc:407] Connection header received
0808 02:25:27.804105 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:27.804109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:27.804185 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:27.804277 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:27.805167 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.805945 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.807341 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.808065 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.808202 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:27.813014 (+ 4812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:35957: BlockingRecv error: recv got EOF from 127.19.231.1:35957 (error 108)
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":674,"threads_started":1}
Aug 08 02:25:27 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:27 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:27.989604 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:27.979172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:33091)
0808 02:25:27.979755 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:27.979771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:27.979941 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:27.980306 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:27.980311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:27.980746 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:27.980938 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:27.980948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.981973 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.981979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:27.983112 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:27.983124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.983839 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.983844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:27.984022 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:27.984760 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:27.984777 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:27.989211 (+ 4434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":322,"threads_started":1}
W20260808 02:25:27.989689 30099 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:27.979204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:33091 (local address 127.19.231.62:45809)
0808 02:25:27.979402 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:27.979410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:27.979877 (+ 467us) server_negotiation.cc:407] Connection header received
0808 02:25:27.980075 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:27.980082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:27.980161 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:27.980275 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:27.981094 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.981838 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.983262 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:27.983681 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:27.983810 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:27.989432 (+ 5622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:33091: BlockingRecv error: recv got EOF from 127.19.231.3:33091 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 02:25:28 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:28 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:28.053247 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:28.042293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:35417)
0808 02:25:28.042901 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:28.042918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:28.043028 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:28.043433 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:28.043436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:28.043889 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:28.044040 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:28.044049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.045715 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.045719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:28.046757 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:28.046771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.047955 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.047959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:28.048165 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:28.048996 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:28.049018 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:28.053011 (+ 3993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":352,"threads_started":1}
W20260808 02:25:28.053838 30099 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:28.042442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:35417 (local address 127.19.231.62:45809)
0808 02:25:28.042704 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:28.042712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:28.043003 (+ 291us) server_negotiation.cc:407] Connection header received
0808 02:25:28.043154 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:28.043161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:28.043281 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:28.043409 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:28.044188 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.045569 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.046894 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.047808 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.047965 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:28.053314 (+ 5349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:35417: BlockingRecv error: recv got EOF from 127.19.231.2:35417 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 08 02:25:28 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:28 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:28.827652 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:28.813866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:59227)
0808 02:25:28.814488 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:28.814506 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:28.814682 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:28.815030 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:28.815035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:28.815522 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:28.815727 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:28.815737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.816818 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.816822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:28.817846 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:28.817857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.818581 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.818587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:28.818791 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:28.823507 (+ 4716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:28.823527 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:28.827308 (+ 3781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":352,"threads_started":1}
W20260808 02:25:28.827816 30104 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:28.813949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:59227 (local address 127.19.231.62:45809)
0808 02:25:28.814605 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:28.814663 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:28.814687 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:25:28.814810 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:28.814813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:28.814878 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:28.814971 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:28.815892 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.816668 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.817993 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.818392 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.818487 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:28.827548 (+ 9061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:59227: BlockingRecv error: recv got EOF from 127.19.231.1:59227 (error 108)
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":398,"threads_started":1}
Aug 08 02:25:28 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:28 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:28.999848 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:28.990964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:39767)
0808 02:25:28.991580 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:28.991596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:28.991711 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:28.991983 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:28.991986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:28.992499 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:28.992654 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:28.992663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.993679 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.993684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:28.994688 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:28.994701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.995370 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.995374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:28.995511 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:28.996172 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:28.996189 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:28.999613 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":324,"threads_started":1}
W20260808 02:25:29.000005 30104 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:28.991093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:39767 (local address 127.19.231.62:45809)
0808 02:25:28.991286 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:28.991292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:28.991675 (+ 383us) server_negotiation.cc:407] Connection header received
0808 02:25:28.991793 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:28.991797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:28.991860 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:28.991937 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:28.992783 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.993535 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.994827 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:28.995233 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:28.995324 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:28.999791 (+ 4467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:39767: BlockingRecv error: recv got EOF from 127.19.231.3:39767 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 02:25:29 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:29 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:29.063328 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:29.054206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:34153)
0808 02:25:29.054780 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:29.054798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:29.054910 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:29.055258 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:29.055262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:29.055686 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:29.055865 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:29.055874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.056906 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.056910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:29.057878 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:29.057890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.058542 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.058547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:29.058886 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:29.059534 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:29.059559 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:29.063007 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":322,"threads_started":1}
W20260808 02:25:29.063540 30104 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:29.054265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:34153 (local address 127.19.231.62:45809)
0808 02:25:29.054557 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:29.054561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:29.054873 (+ 312us) server_negotiation.cc:407] Connection header received
0808 02:25:29.055022 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:29.055025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:29.055092 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:29.055180 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:29.056002 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.056764 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.058000 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.058377 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.058468 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:29.063258 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:34153: BlockingRecv error: recv got EOF from 127.19.231.2:34153 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Aug 08 02:25:29 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:29 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:29.838181 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:29.828678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:47075)
0808 02:25:29.829289 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:29.829306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:29.829418 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:29.829723 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:29.829727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:29.830182 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:29.830363 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:29.830372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.831437 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.831441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:29.832515 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:29.832528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.833215 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.833219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:29.833334 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:29.833996 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:29.834015 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:29.837788 (+ 3773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":275,"threads_started":1}
W20260808 02:25:29.838320 30107 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:29.828720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:47075 (local address 127.19.231.62:45809)
0808 02:25:29.829264 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:29.829268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:29.829381 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:25:29.829517 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:29.829520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:29.829582 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:29.829683 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:29.830492 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.831298 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.832646 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:29.833069 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:29.833171 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:29.838034 (+ 4863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:47075: BlockingRecv error: recv got EOF from 127.19.231.1:47075 (error 108)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":298,"threads_started":1}
Aug 08 02:25:30 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:30 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:30.011436 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:30.000876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:60371)
0808 02:25:30.001457 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:30.001473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:30.001654 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:30.002068 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:30.002073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:30.002592 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:30.002897 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:30.002913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.004249 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.004256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:30.005718 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:30.005735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.006701 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.006707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:30.006902 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:30.007530 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:30.007547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:30.011056 (+ 3509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":300,"threads_started":1}
W20260808 02:25:30.011487 30107 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:30.001019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:60371 (local address 127.19.231.62:45809)
0808 02:25:30.001276 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:30.001282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:30.001570 (+ 288us) server_negotiation.cc:407] Connection header received
0808 02:25:30.001791 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:30.001798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:30.001902 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:30.002020 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:30.003072 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.004077 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.005863 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.006470 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.006619 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:30.011265 (+ 4646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:60371: BlockingRecv error: recv got EOF from 127.19.231.3:60371 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 02:25:30 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:30 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:30.074394 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:30.064586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:38735)
0808 02:25:30.065284 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:30.065303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:30.065484 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:30.065891 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:30.065896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:30.066505 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:30.066805 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:30.066822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.067946 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.067952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:30.068967 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:30.068979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.069710 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.069715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:30.069888 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:30.070485 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:30.070503 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:30.074016 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":339,"threads_started":1}
W20260808 02:25:30.074476 30107 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:30.064580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:38735 (local address 127.19.231.62:45809)
0808 02:25:30.064850 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:30.064855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:30.065408 (+ 553us) server_negotiation.cc:407] Connection header received
0808 02:25:30.065657 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:30.065662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:30.065724 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:30.065855 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:30.066966 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.067796 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.069113 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.069544 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.069686 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:30.074215 (+ 4529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:38735: BlockingRecv error: recv got EOF from 127.19.231.2:38735 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Aug 08 02:25:30 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:30 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:30.849267 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:30.839519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:38505)
0808 02:25:30.840215 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:30.840240 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:30.840387 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:30.840837 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:30.840841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:30.841251 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:30.841432 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:30.841442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.842539 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.842578 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:30.843522 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:30.843534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.844277 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.844282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:30.844431 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:30.845072 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:30.845096 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:30.848917 (+ 3821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":298,"threads_started":1}
W20260808 02:25:30.849417 30112 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:30.839696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:38505 (local address 127.19.231.62:45809)
0808 02:25:30.840519 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:30.840526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:30.840544 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:25:30.840601 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:30.840606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:30.840685 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:30.840775 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:30.841598 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.842378 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.843660 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:30.844072 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:30.844172 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:30.849122 (+ 4950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:38505: BlockingRecv error: recv got EOF from 127.19.231.1:38505 (error 108)
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":534,"threads_started":1}
Aug 08 02:25:31 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:31 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:31.021766 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:31.012473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:49681)
0808 02:25:31.013159 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:31.013176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:31.013291 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:31.013646 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:31.013649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:31.014068 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:31.014237 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:31.014245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.015285 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.015288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:31.016268 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:31.016280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.016949 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.016953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:31.017077 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:31.017683 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:31.017701 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:31.021406 (+ 3705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":289,"threads_started":1}
W20260808 02:25:31.021849 30112 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:31.012599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:49681 (local address 127.19.231.62:45809)
0808 02:25:31.012789 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:31.012793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:31.013253 (+ 460us) server_negotiation.cc:407] Connection header received
0808 02:25:31.013392 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:31.013396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:31.013460 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:31.013543 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:31.014361 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.015145 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.016397 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.016813 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.016907 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:31.021620 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:49681: BlockingRecv error: recv got EOF from 127.19.231.3:49681 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 02:25:31 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:31 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:31.085309 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:31.075610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:52851)
0808 02:25:31.076206 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:31.076223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:31.076336 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:31.076671 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:31.076675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:31.077140 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:31.077312 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:31.077321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.078491 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.078495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:31.079649 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:31.079662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.080504 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.080510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:31.080637 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:31.081428 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:31.081446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:31.085041 (+ 3595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":298,"threads_started":1}
W20260808 02:25:31.085701 30112 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:31.075674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:52851 (local address 127.19.231.62:45809)
0808 02:25:31.075842 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:31.075847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:31.076302 (+ 455us) server_negotiation.cc:407] Connection header received
0808 02:25:31.076447 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:31.076450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:31.076511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:31.076596 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:31.077444 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.078352 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.079788 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.080311 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.080405 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:31.085273 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:52851: BlockingRecv error: recv got EOF from 127.19.231.2:52851 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 02:25:31 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:31 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:31.860307 30115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:31.850295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:35429)
0808 02:25:31.850887 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:31.850905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:31.851010 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:31.851390 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:31.851394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:31.851884 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:31.852068 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:31.852080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.853224 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.853232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:31.854665 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:31.854687 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.855586 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.855591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:31.855719 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:31.856285 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:31.856304 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:31.860072 (+ 3768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
W20260808 02:25:31.860528 30116 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:31.850414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:35429 (local address 127.19.231.62:45809)
0808 02:25:31.850975 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:31.850979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:31.850997 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:25:31.851118 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:31.851122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:31.851198 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:31.851290 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:31.852217 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.853026 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.854839 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:31.855430 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:31.855528 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:31.860276 (+ 4748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:35429: BlockingRecv error: recv got EOF from 127.19.231.1:35429 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1}
Aug 08 02:25:32 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:32 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:32.032639 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:32.022875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:53799)
0808 02:25:32.023485 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:32.023502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:32.023715 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:32.024088 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:32.024093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:32.024535 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:32.024712 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:32.024721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.025744 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.025767 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:32.026773 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:32.026784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.027519 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.027524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:32.027698 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:32.028286 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:32.028303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:32.032213 (+ 3910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":314,"threads_started":1}
W20260808 02:25:32.032704 30116 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:32.023027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:53799 (local address 127.19.231.62:45809)
0808 02:25:32.023258 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:32.023264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:32.023640 (+ 376us) server_negotiation.cc:407] Connection header received
0808 02:25:32.023850 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:32.023860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:32.023932 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:32.024040 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:32.024860 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.025598 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.026914 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.027333 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.027478 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:32.032439 (+ 4961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:53799: BlockingRecv error: recv got EOF from 127.19.231.3:53799 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 08 02:25:32 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:32 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:32.095554 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:32.086367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:59893)
0808 02:25:32.087009 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:32.087026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:32.087135 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:32.087436 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:32.087439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:32.087874 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:32.088018 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:32.088026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.089103 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.089106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:32.090062 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:32.090073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.090893 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.090898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:32.091024 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:32.091701 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:32.091718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:32.095276 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":405,"threads_started":1}
W20260808 02:25:32.096092 30116 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:32.086414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:59893 (local address 127.19.231.62:45809)
0808 02:25:32.086669 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:32.086680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:32.087118 (+ 438us) server_negotiation.cc:407] Connection header received
0808 02:25:32.087223 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:32.087227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:32.087308 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:32.087398 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:32.088151 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.088974 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.090214 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.090745 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.090844 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:32.095597 (+ 4753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:59893: BlockingRecv error: recv got EOF from 127.19.231.2:59893 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 02:25:32 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:32 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:32.871650 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:32.861257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:39843)
0808 02:25:32.861783 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:32.861801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:32.861913 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:32.862534 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:32.862541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:32.863012 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:32.863201 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:32.863212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.864475 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.864489 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:32.865562 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:32.865593 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.866420 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.866427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:32.866670 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:32.867385 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:32.867406 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:32.871248 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
W20260808 02:25:32.871943 30120 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:32.861393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:39843 (local address 127.19.231.62:45809)
0808 02:25:32.862220 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:32.862225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:32.862240 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:25:32.862286 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:32.862289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:32.862346 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:32.862436 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:32.863412 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.864291 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.865751 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:32.866244 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:32.866400 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:32.871657 (+ 5257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:39843: BlockingRecv error: recv got EOF from 127.19.231.1:39843 (error 108)
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":572,"threads_started":1}
Aug 08 02:25:33 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:33 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:33.043704 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:33.033885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:39599)
0808 02:25:33.034465 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:33.034480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:33.034664 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:33.034995 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:33.034999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:33.035478 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:33.035660 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:33.035670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.036701 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.036707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:33.037709 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:33.037722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.038634 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.038641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:33.038785 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:33.039498 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:33.039518 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:33.043303 (+ 3785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":277,"threads_started":1}
W20260808 02:25:33.043746 30120 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:33.033945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:39599 (local address 127.19.231.62:45809)
0808 02:25:33.034266 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:33.034275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:33.034568 (+ 293us) server_negotiation.cc:407] Connection header received
0808 02:25:33.034797 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:33.034801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:33.034863 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:33.034963 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:33.035801 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.036548 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.037845 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.038265 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.038427 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:33.043513 (+ 5086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:39599: BlockingRecv error: recv got EOF from 127.19.231.3:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 02:25:33 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:33 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:33.106289 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:33.096493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:35149)
0808 02:25:33.097067 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:33.097084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:33.097250 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:33.097616 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:33.097620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:33.098029 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:33.098213 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:33.098223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.099418 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.099426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:33.100459 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:33.100471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.101242 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.101248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:33.101415 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:33.102108 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:33.102126 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:33.105932 (+ 3806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":320,"threads_started":1}
W20260808 02:25:33.106523 30120 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:33.096586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:35149 (local address 127.19.231.62:45809)
0808 02:25:33.096865 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:33.096874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:33.097184 (+ 310us) server_negotiation.cc:407] Connection header received
0808 02:25:33.097383 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:33.097388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:33.097470 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:33.097589 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:33.098373 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.099264 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.100605 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.101089 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.101227 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:33.106180 (+ 4953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:35149: BlockingRecv error: recv got EOF from 127.19.231.2:35149 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 08 02:25:33 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:33 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:33.887117 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:33.872773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:38409)
0808 02:25:33.873351 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:33.873376 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:33.873478 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:33.873772 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:33.873775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:33.874184 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:33.874358 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:33.874366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.875543 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.875547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:33.876470 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:33.876481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.877222 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.877226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:33.877342 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:33.883239 (+ 5897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:33.883259 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:33.886885 (+ 3626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":297,"threads_started":1}
W20260808 02:25:33.887548 30124 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:33.872901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:38409 (local address 127.19.231.62:45809)
0808 02:25:33.873492 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:33.873497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:33.873514 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:25:33.873566 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:33.873571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:33.873636 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:33.873734 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:33.874505 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.875418 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.876615 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:33.877092 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:33.877193 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:33.887187 (+ 9994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:38409: BlockingRecv error: recv got EOF from 127.19.231.1:38409 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":326,"threads_started":1}
Aug 08 02:25:34 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:34 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:34.054492 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:34.044652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:37773)
0808 02:25:34.045243 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:34.045261 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:34.045384 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:34.045704 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:34.045707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:34.046136 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:34.046313 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:34.046322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.047904 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.047910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:34.049126 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:34.049141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.049886 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.049892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:34.050064 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:34.050731 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:34.050750 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:34.054219 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":327,"threads_started":1}
W20260808 02:25:34.054611 30124 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:34.044725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:37773 (local address 127.19.231.62:45809)
0808 02:25:34.044892 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:34.044896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:34.045337 (+ 441us) server_negotiation.cc:407] Connection header received
0808 02:25:34.045480 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:34.045483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:34.045545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:34.045628 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:34.046440 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.047739 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.049292 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.049698 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.049830 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:34.054404 (+ 4574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:37773: BlockingRecv error: recv got EOF from 127.19.231.3:37773 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 02:25:34 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:34 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:34.116374 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:34.107486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:54703)
0808 02:25:34.108239 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:34.108256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:34.108398 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:34.108808 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:34.108814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:34.109268 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:34.109469 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:34.109478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.110466 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.110470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:34.111387 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:34.111398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.112038 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.112042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:34.112173 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:34.112744 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:34.112762 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:34.116143 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":507,"threads_started":1}
W20260808 02:25:34.116605 30124 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:34.107617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:54703 (local address 127.19.231.62:45809)
0808 02:25:34.107825 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:34.107831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:34.108345 (+ 514us) server_negotiation.cc:407] Connection header received
0808 02:25:34.108527 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:34.108531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:34.108637 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:34.108742 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:34.109611 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.110356 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.111519 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.111913 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.112008 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:34.116356 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:54703: BlockingRecv error: recv got EOF from 127.19.231.2:54703 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 08 02:25:34 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:34 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:34.897410 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:34.888127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:51213)
0808 02:25:34.888685 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:34.888710 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:34.888812 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:34.889098 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:34.889101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:34.889515 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:34.889699 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:34.889708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.890745 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.890748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:34.891706 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:34.891718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.892394 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.892398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:34.892530 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:34.893212 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:34.893231 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:34.897064 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":290,"threads_started":1}
W20260808 02:25:34.897608 30128 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:34.888166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:51213 (local address 127.19.231.62:45809)
0808 02:25:34.888756 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:34.888760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:34.888777 (+ 17us) server_negotiation.cc:407] Connection header received
0808 02:25:34.888919 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:34.888923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:34.888989 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:34.889085 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:34.889852 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.890635 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.891834 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:34.892270 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:34.892376 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:34.897328 (+ 4952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:51213: BlockingRecv error: recv got EOF from 127.19.231.1:51213 (error 108)
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":307,"threads_started":1}
Aug 08 02:25:35 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:35 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:35.065714 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:35.055540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:56281)
0808 02:25:35.056124 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:35.056142 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:35.056344 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:35.056776 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:35.056782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:35.057288 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:35.057493 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:35.057504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.058682 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.058686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:35.059680 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:35.059692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.060479 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.060485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:35.060690 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:35.061316 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:35.061335 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:35.065342 (+ 4007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":302,"threads_started":1}
W20260808 02:25:35.065834 30128 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:35.055687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:56281 (local address 127.19.231.62:45809)
0808 02:25:35.055932 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:35.055940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:35.056257 (+ 317us) server_negotiation.cc:407] Connection header received
0808 02:25:35.056486 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:35.056498 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:35.056616 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:35.056739 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:35.057658 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.058473 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.059842 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.060303 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.060461 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:35.065559 (+ 5098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:56281: BlockingRecv error: recv got EOF from 127.19.231.3:56281 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 08 02:25:35 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:35 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:35.127286 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:35.117392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:44109)
0808 02:25:35.118148 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:35.118167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:35.118292 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:35.118635 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:35.118639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:35.119143 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:35.119335 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:35.119345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.120368 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.120373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:35.121407 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:35.121421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.122141 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.122146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:35.122312 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:35.122975 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:35.122995 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:35.127040 (+ 4045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":434,"threads_started":1}
W20260808 02:25:35.127627 30128 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:35.117451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:44109 (local address 127.19.231.62:45809)
0808 02:25:35.117619 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:35.117624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:35.118246 (+ 622us) server_negotiation.cc:407] Connection header received
0808 02:25:35.118395 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:35.118398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:35.118474 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:35.118555 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:35.119470 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.120219 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.121538 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.121958 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.122046 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:35.127262 (+ 5216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:44109: BlockingRecv error: recv got EOF from 127.19.231.2:44109 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 02:25:35 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:35 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:35.908301 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:35.898498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:50117)
0808 02:25:35.899259 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:35.899307 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:35.899417 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:35.899901 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:35.899905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:35.900374 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:35.900610 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:35.900622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.901694 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.901703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:35.902720 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:35.902733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.903538 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.903542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:35.903699 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:35.904398 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:35.904416 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:35.908063 (+ 3647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":488,"threads_started":1}
W20260808 02:25:35.908656 30132 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:35.898657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:50117 (local address 127.19.231.62:45809)
0808 02:25:35.899623 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:35.899628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:35.899646 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:25:35.899691 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:35.899695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:35.899752 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:35.899841 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:35.900748 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.901530 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.902859 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:35.903387 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:35.903501 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:35.908262 (+ 4761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:50117: BlockingRecv error: recv got EOF from 127.19.231.1:50117 (error 108)
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":727,"threads_started":1}
Aug 08 02:25:36 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:36 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:36.077119 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:36.066933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:41073)
0808 02:25:36.067549 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:36.067567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:36.067702 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:36.068138 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:36.068143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:36.068805 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:36.069070 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:36.069082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.070681 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.070688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:36.071765 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:36.071779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.072527 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.072531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:36.072658 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:36.073286 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:36.073303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:36.076876 (+ 3573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":350,"threads_started":1}
W20260808 02:25:36.077371 30132 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:36.067097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:41073 (local address 127.19.231.62:45809)
0808 02:25:36.067286 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:36.067290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:36.067681 (+ 391us) server_negotiation.cc:407] Connection header received
0808 02:25:36.067828 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:36.067834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:36.067925 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:36.068065 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:36.069243 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.070472 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.071912 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.072398 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.072505 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:36.077126 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:41073: BlockingRecv error: recv got EOF from 127.19.231.3:41073 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 02:25:36 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:36 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:36.138303 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:36.128356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:60033)
0808 02:25:36.128968 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:36.128984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:36.129102 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:36.129484 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:36.129488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:36.130075 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:36.130276 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:36.130290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.131602 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.131606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:36.132539 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:36.132552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.133200 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.133205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:36.133359 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:36.134207 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:36.134228 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:36.138054 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":364,"threads_started":1}
W20260808 02:25:36.138499 30132 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:36.128510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:60033 (local address 127.19.231.62:45809)
0808 02:25:36.128716 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:36.128721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:36.129074 (+ 353us) server_negotiation.cc:407] Connection header received
0808 02:25:36.129231 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:36.129238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:36.129334 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:36.129460 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:36.130431 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.131424 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.132678 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.133069 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.133162 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:36.138265 (+ 5103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:60033: BlockingRecv error: recv got EOF from 127.19.231.2:60033 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 02:25:36 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:36 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.1@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:36.919852 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:36.909433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.1:35007)
0808 02:25:36.909951 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:36.909970 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:36.910082 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:36.910566 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:36.910570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:36.911062 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:36.911257 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:36.911269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.912612 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.912616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:36.913689 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:36.913703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.914379 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.914383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:36.914522 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:36.915434 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:36.915457 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:36.919560 (+ 4103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
W20260808 02:25:36.920367 30137 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:36.909583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:35007 (local address 127.19.231.62:45809)
0808 02:25:36.910298 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:36.910302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:36.910318 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:25:36.910361 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:36.910365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:36.910419 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:36.910505 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:36.911414 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.912442 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.913831 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:36.914242 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:36.914338 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:36.919820 (+ 5482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.1:35007: BlockingRecv error: recv got EOF from 127.19.231.1:35007 (error 108)
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":475,"threads_started":1}
Aug 08 02:25:37 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:37 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.3@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:37.087338 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:37.078104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.3:37517)
0808 02:25:37.078660 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:37.078683 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:37.078836 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:37.079179 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:37.079183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:37.079630 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:37.079792 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:37.079801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.080896 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.080901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:37.081807 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:37.081819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.082511 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.082517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:37.082725 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:37.083394 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:37.083412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:37.087087 (+ 3675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
W20260808 02:25:37.087622 30137 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:37.078170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:37517 (local address 127.19.231.62:45809)
0808 02:25:37.078468 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:37.078474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:37.078761 (+ 287us) server_negotiation.cc:407] Connection header received
0808 02:25:37.078975 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:37.078979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:37.079042 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:37.079145 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:37.079939 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.080750 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.081946 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.082358 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.082489 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:37.087293 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.3:37517: BlockingRecv error: recv got EOF from 127.19.231.3:37517 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Aug 08 02:25:37 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 02:25:37 dist-test-slave-pgkr krb5kdc[29041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.231.2@KRBTEST.COM for oryx/127.19.231.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 02:25:37.149984 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:37.139705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45809 (local address 127.19.231.2:56961)
0808 02:25:37.140290 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:37.140309 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:37.140479 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:37.140886 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:37.140891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:37.141347 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:37.141605 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:37.141617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.142816 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.142822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:37.144086 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:37.144100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.145017 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.145025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:37.145236 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:37.145930 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:37.145950 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:37.149744 (+ 3794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.231.62:45809: Server oryx/127.19.231.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":297,"threads_started":1}
W20260808 02:25:37.150190 30137 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:37.139807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:56961 (local address 127.19.231.62:45809)
0808 02:25:37.140098 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:37.140107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:37.140401 (+ 294us) server_negotiation.cc:407] Connection header received
0808 02:25:37.140640 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:37.140644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:37.140737 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:37.140853 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:37.141774 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.142602 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.144252 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.144842 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.144984 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:37.149938 (+ 4954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.231.2:56961: BlockingRecv error: recv got EOF from 127.19.231.2:56961 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 02:25:37.703697 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29649
I20260808 02:25:37.716796 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:37.705081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.19.231.1:46799)
0808 02:25:37.705843 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:37.705850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:37.705876 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:25:37.705942 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:37.705948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:37.706204 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:37.706371 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:37.708071 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.709147 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.710328 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.710939 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.711045 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:37.713576 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:37.713585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:37.713598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:37.713638 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:37.716272 (+ 2634us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.231.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 02:25:37.716569 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46430: Request ticket server kudu/127.19.231.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":431,"threads_started":1}
W20260808 02:25:37.716919 30141 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46430: Request ticket server kudu/127.19.231.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 02:25:37.716995 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29649: Not authorized: Client connection negotiation failed: client connection to 127.19.231.1:46799: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.231.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 02:25:37.717248 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29649
I20260808 02:25:37.739537 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29804
I20260808 02:25:37.752890 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:37.740606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.19.231.2:38305)
0808 02:25:37.741285 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:37.741295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:37.741326 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:25:37.741402 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:37.741410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:37.741714 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:37.741935 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:37.743430 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.744684 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.745938 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.746597 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.746760 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:37.749513 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:37.749521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:37.749534 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:37.749589 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:37.752385 (+ 2796us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.231.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 02:25:37.752678 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41950: Request ticket server kudu/127.19.231.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":326,"threads_started":1}
W20260808 02:25:37.753017 30142 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41950: Request ticket server kudu/127.19.231.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 02:25:37.753093 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29804: Not authorized: Client connection negotiation failed: client connection to 127.19.231.2:38305: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.231.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 02:25:37.753335 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29804
I20260808 02:25:37.774422 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29942
I20260808 02:25:37.788542 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:37.775776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.19.231.3:41469)
0808 02:25:37.776394 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:37.776405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:37.776632 (+ 227us) server_negotiation.cc:407] Connection header received
0808 02:25:37.776829 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:37.776839 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:37.777198 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:37.777413 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:37.778882 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.780465 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.781913 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:37.782492 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:37.782600 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:37.785283 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:37.785292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:37.785305 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:37.785341 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:37.788030 (+ 2689us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.231.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 02:25:37.788310 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58584: Request ticket server kudu/127.19.231.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":281,"threads_started":1}
W20260808 02:25:37.788661 30143 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58584: Request ticket server kudu/127.19.231.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 02:25:37.788771 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29942: Not authorized: Client connection negotiation failed: client connection to 127.19.231.3:41469: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.231.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 02:25:37.788975 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29942
I20260808 02:25:37.810475 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 29577
I20260808 02:25:37.812037 29641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:37.933432 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 29577
2026-08-08T02:25:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22017 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T02:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:38Z Disabled control of system clock
I20260808 02:25:38.030300 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:35285
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:40841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 02:25:38.378410 30148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:38.378808 30148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:38.378979 30148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:38.386114 30148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:38.386209 30148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:38.386298 30148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:38.386404 30148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:38.386489 30148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:38.395697 30148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:35285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:38.396970 30148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:38.398696 30148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:38.411255 30154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:38.411213 30156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:38.411299 30153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:38.415993 30148 server_base.cc:1034] running on GCE node
I20260808 02:25:38.416792 30148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:38.418143 30148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:38.419338 30148 hybrid_clock.cc:648] HybridClock initialized: now 1786155938419305 us; error 32 us; skew 500 ppm
I20260808 02:25:38.419646 30148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:38.421917 30148 webserver.cc:460] Webserver started at http://127.19.231.62:43521/ using document root <none> and password file <none>
I20260808 02:25:38.422570 30148 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:38.422742 30148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:38.423036 30148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:38.425938 30148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "86a915d4cfc649c7b32e7d9ce1e70700"
format_stamp: "Formatted at 2026-08-08 02:25:38 on dist-test-slave-pgkr"
server_key: "6b15b81ef3d6500fd64299a08382f422"
server_key_iv: "cd56b5892483b46b143cf3f89fa928a2"
server_key_version: "encryptionkey@0"
I20260808 02:25:38.426775 30148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "86a915d4cfc649c7b32e7d9ce1e70700"
format_stamp: "Formatted at 2026-08-08 02:25:38 on dist-test-slave-pgkr"
server_key: "6b15b81ef3d6500fd64299a08382f422"
server_key_iv: "cd56b5892483b46b143cf3f89fa928a2"
server_key_version: "encryptionkey@0"
I20260808 02:25:38.431981 30148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:25:38.435595 30163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:38.436990 30148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:25:38.437201 30148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "86a915d4cfc649c7b32e7d9ce1e70700"
format_stamp: "Formatted at 2026-08-08 02:25:38 on dist-test-slave-pgkr"
server_key: "6b15b81ef3d6500fd64299a08382f422"
server_key_iv: "cd56b5892483b46b143cf3f89fa928a2"
server_key_version: "encryptionkey@0"
I20260808 02:25:38.437425 30148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:38.455860 30148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:38.456709 30148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:38.456996 30148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:38.475594 30148 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:35285
I20260808 02:25:38.475628 30215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:35285 every 8 connection(s)
I20260808 02:25:38.476985 30148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:38.480840 30216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:38.486406 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 30148
I20260808 02:25:38.486573 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:25:38.487337 20380 external_mini_cluster.cc:1428] Setting key 413f9234d9fc7a25fc68b38aa9a8de08
I20260808 02:25:38.492575 30216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700: Bootstrap starting.
I20260808 02:25:38.496802 30216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:38.498013 30216 log.cc:824] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:38.501279 30216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700: No bootstrap required, opened a new log
I20260808 02:25:38.502141 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:38.489510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.19.231.62:35285)
0808 02:25:38.490713 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:38.490724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:38.490767 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:25:38.492409 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:38.492430 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:38.492861 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:38.493232 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:38.493933 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:38.495576 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:38.497543 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:38.498269 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:38.498405 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:38.499841 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:38.499848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:38.499861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:38.500413 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:38.500485 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:38.500504 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:38.500591 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:38.500830 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 02:25:38.501185 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":464,"threads_started":1}
I20260808 02:25:38.508925 30216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35285 } }
I20260808 02:25:38.509313 30216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:38.509436 30216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a915d4cfc649c7b32e7d9ce1e70700, State: Initialized, Role: FOLLOWER
I20260808 02:25:38.509982 30216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35285 } }
I20260808 02:25:38.510210 30216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:38.510356 30216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:38.510533 30216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:38.512892 30216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35285 } }
I20260808 02:25:38.513339 30216 leader_election.cc:302] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a915d4cfc649c7b32e7d9ce1e70700; no voters:
I20260808 02:25:38.513855 30216 leader_election.cc:288] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:38.514029 30221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:38.514807 30221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [term 1 LEADER]: Becoming Leader. State: Replica: 86a915d4cfc649c7b32e7d9ce1e70700, State: Running, Role: LEADER
I20260808 02:25:38.515333 30221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35285 } }
I20260808 02:25:38.515923 30216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:38.518340 30222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35285 } } }
I20260808 02:25:38.518604 30223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a915d4cfc649c7b32e7d9ce1e70700. Latest consensus state: current_term: 1 leader_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a915d4cfc649c7b32e7d9ce1e70700" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 35285 } } }
I20260808 02:25:38.518838 30222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:38.518910 30223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:38.523761 30226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:38.530542 30226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:38.539397 30226 catalog_manager.cc:1347] Generated new cluster ID: 7b95888848af4a18907bb4fdc1ed794e
I20260808 02:25:38.539531 30226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:38.554694 30226 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:38.556490 30226 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:38.566226 30226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a915d4cfc649c7b32e7d9ce1e70700: Generated new TSK 0
I20260808 02:25:38.567299 30226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:25:38.576263 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35285
--builtin_ntp_servers=127.19.231.20:40841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 02:25:38.918047 30240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:38.918385 30240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:38.918551 30240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:38.925930 30240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:38.926123 30240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:38.926312 30240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:25:38.935516 30240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:38.936801 30240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:38.938941 30240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:38.952301 30246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:38.952245 30247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:38.952176 30249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:38.952996 30240 server_base.cc:1034] running on GCE node
I20260808 02:25:38.953579 30240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:38.954496 30240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:38.955703 30240 hybrid_clock.cc:648] HybridClock initialized: now 1786155938955658 us; error 38 us; skew 500 ppm
I20260808 02:25:38.956019 30240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:38.963701 30240 webserver.cc:460] Webserver started at http://127.19.231.1:40779/ using document root <none> and password file <none>
I20260808 02:25:38.964474 30240 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:38.964571 30240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:38.964946 30240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:38.968014 30240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "b4d4f33630b743eca9cdb104d326269f"
format_stamp: "Formatted at 2026-08-08 02:25:38 on dist-test-slave-pgkr"
server_key: "b5622e8ac3b0ace7d3804146a7b991f4"
server_key_iv: "10f06b18a1e927a3941493424d34c51f"
server_key_version: "encryptionkey@0"
I20260808 02:25:38.968806 30240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "b4d4f33630b743eca9cdb104d326269f"
format_stamp: "Formatted at 2026-08-08 02:25:38 on dist-test-slave-pgkr"
server_key: "b5622e8ac3b0ace7d3804146a7b991f4"
server_key_iv: "10f06b18a1e927a3941493424d34c51f"
server_key_version: "encryptionkey@0"
I20260808 02:25:38.973939 30240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 02:25:38.977519 30255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:38.978888 30240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:25:38.979058 30240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "b4d4f33630b743eca9cdb104d326269f"
format_stamp: "Formatted at 2026-08-08 02:25:38 on dist-test-slave-pgkr"
server_key: "b5622e8ac3b0ace7d3804146a7b991f4"
server_key_iv: "10f06b18a1e927a3941493424d34c51f"
server_key_version: "encryptionkey@0"
I20260808 02:25:38.979282 30240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:38.995178 30240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:38.996049 30240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:38.996372 30240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:38.997318 30240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:38.998811 30240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:38.998935 30240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:38.999073 30240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:38.999171 30240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:39.034000 30240 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:40721
I20260808 02:25:39.034101 30373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:40721 every 8 connection(s)
I20260808 02:25:39.035386 30240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:39.037247 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 30240
I20260808 02:25:39.037456 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:25:39.038177 20380 external_mini_cluster.cc:1428] Setting key 9f4804a0e99a86cdf9aa6b6c8d93bbde
I20260808 02:25:39.041814 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35285
--builtin_ntp_servers=127.19.231.20:40841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:25:39.051381 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:39.038257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:56745 (local address 127.19.231.62:35285)
0808 02:25:39.038770 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:39.038776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:39.040187 (+ 1411us) server_negotiation.cc:407] Connection header received
0808 02:25:39.043089 (+ 2902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:39.043094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:39.043160 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:39.043255 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:39.044817 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.045517 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.048184 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.048582 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.048689 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:39.050444 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:39.050449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:39.050452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:39.050469 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:39.050514 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:39.050526 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:39.050596 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:39.050941 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 02:25:39.051047 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":231,"threads_started":1}
I20260808 02:25:39.053021 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:39.038973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:35285 (local address 127.19.231.1:56745)
0808 02:25:39.040024 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:39.040073 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:39.042774 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:39.043563 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:39.043579 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:39.043710 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:39.044615 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:39.044647 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.045683 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.045713 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:39.048030 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:39.048051 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.048825 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.048836 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:39.049196 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:39.050020 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:25:39.050026 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:39.050085 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:25:39.050205 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:39.050654 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:39.050668 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:39.050679 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 02:25:39.050897 (+ 218us) client_negotiation.cc:240] Negotiation successful
0808 02:25:39.051413 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":349,"threads_started":1}
I20260808 02:25:39.054607 30374 heartbeater.cc:347] Connected to a master server at 127.19.231.62:35285
I20260808 02:25:39.055035 30374 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:39.056067 30374 heartbeater.cc:511] Master 127.19.231.62:35285 requested a full tablet report, sending...
I20260808 02:25:39.058542 30180 ts_manager.cc:194] Registered new tserver with Master: b4d4f33630b743eca9cdb104d326269f (127.19.231.1:40721)
I20260808 02:25:39.060810 30180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.1:56745
W20260808 02:25:39.389427 30378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:39.389706 30378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:39.389793 30378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:39.396987 30378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:39.397082 30378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:39.397183 30378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:25:39.406162 30378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:39.407385 30378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:39.409158 30378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:39.422429 30386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:39.422504 30384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:39.422461 30383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:39.427860 30378 server_base.cc:1034] running on GCE node
I20260808 02:25:39.428468 30378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:39.429440 30378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:39.430830 30378 hybrid_clock.cc:648] HybridClock initialized: now 1786155939430765 us; error 59 us; skew 500 ppm
I20260808 02:25:39.431205 30378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:39.433635 30378 webserver.cc:460] Webserver started at http://127.19.231.2:34403/ using document root <none> and password file <none>
I20260808 02:25:39.434432 30378 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:39.434556 30378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:39.434980 30378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:39.437985 30378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "a944c9ffa0a6405f9b5b749186c11b79"
format_stamp: "Formatted at 2026-08-08 02:25:39 on dist-test-slave-pgkr"
server_key: "59e45bc6a6f79aa03bd5b2b7927d2899"
server_key_iv: "32689fe56fd9adf4acbd1d51690ad3a3"
server_key_version: "encryptionkey@0"
I20260808 02:25:39.438921 30378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "a944c9ffa0a6405f9b5b749186c11b79"
format_stamp: "Formatted at 2026-08-08 02:25:39 on dist-test-slave-pgkr"
server_key: "59e45bc6a6f79aa03bd5b2b7927d2899"
server_key_iv: "32689fe56fd9adf4acbd1d51690ad3a3"
server_key_version: "encryptionkey@0"
I20260808 02:25:39.444378 30378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:25:39.447875 30392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:39.449261 30378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:25:39.449451 30378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "a944c9ffa0a6405f9b5b749186c11b79"
format_stamp: "Formatted at 2026-08-08 02:25:39 on dist-test-slave-pgkr"
server_key: "59e45bc6a6f79aa03bd5b2b7927d2899"
server_key_iv: "32689fe56fd9adf4acbd1d51690ad3a3"
server_key_version: "encryptionkey@0"
I20260808 02:25:39.449683 30378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:39.473626 30378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:39.474530 30378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:39.474936 30378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:39.476075 30378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:39.477747 30378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:39.477887 30378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:39.478021 30378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:39.478117 30378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:39.513810 30378 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:35677
I20260808 02:25:39.513909 30506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:35677 every 8 connection(s)
I20260808 02:25:39.515251 30378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:39.515537 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 30378
I20260808 02:25:39.515710 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:25:39.516399 20380 external_mini_cluster.cc:1428] Setting key 73ce71ec8cddb08a11ff989db85702b3
I20260808 02:25:39.520044 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35285
--builtin_ntp_servers=127.19.231.20:40841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 02:25:39.528796 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:39.518331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:52949 (local address 127.19.231.62:35285)
0808 02:25:39.518537 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:39.518543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:39.520174 (+ 1631us) server_negotiation.cc:407] Connection header received
0808 02:25:39.522080 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:39.522086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:39.522172 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:39.522292 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:39.523452 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.524193 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.525910 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.526306 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.526446 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:39.527865 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:39.527870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:39.527873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:39.527891 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:39.527922 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:39.527933 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:39.528029 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:39.528367 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 02:25:39.528468 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:25:39.529819 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:39.519024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:35285 (local address 127.19.231.2:52949)
0808 02:25:39.520004 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:39.520051 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:39.521809 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:39.522526 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:39.522535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:39.522689 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:39.523281 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:39.523298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.524370 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.524374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:39.525757 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:39.525770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:39.526535 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:39.526542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:39.526823 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:39.527540 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:25:39.527543 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:39.527584 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:25:39.527665 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:39.528029 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:39.528112 (+ 83us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:39.528122 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:25:39.528337 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 02:25:39.528746 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":321,"threads_started":1}
I20260808 02:25:39.531384 30507 heartbeater.cc:347] Connected to a master server at 127.19.231.62:35285
I20260808 02:25:39.531738 30507 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:39.532765 30507 heartbeater.cc:511] Master 127.19.231.62:35285 requested a full tablet report, sending...
I20260808 02:25:39.534315 30180 ts_manager.cc:194] Registered new tserver with Master: a944c9ffa0a6405f9b5b749186c11b79 (127.19.231.2:35677)
I20260808 02:25:39.535300 30180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.2:52949
W20260808 02:25:39.879218 30510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:39.879577 30510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:39.879745 30510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:39.887365 30510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:39.887511 30510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:39.887710 30510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:25:39.896885 30510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:39.898334 30510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:39.900252 30510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:39.913707 30515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:39.913754 30516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:39.913781 30518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:39.914812 30510 server_base.cc:1034] running on GCE node
I20260808 02:25:39.915412 30510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:39.916445 30510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:39.917685 30510 hybrid_clock.cc:648] HybridClock initialized: now 1786155939917649 us; error 58 us; skew 500 ppm
I20260808 02:25:39.918004 30510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:39.920750 30510 webserver.cc:460] Webserver started at http://127.19.231.3:38795/ using document root <none> and password file <none>
I20260808 02:25:39.921514 30510 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:39.921665 30510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:39.922021 30510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:39.925055 30510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "5c7e21c0157542a482e7f630eb1ef61c"
format_stamp: "Formatted at 2026-08-08 02:25:39 on dist-test-slave-pgkr"
server_key: "2b9bee54ef37ef572ff09e9a752c010a"
server_key_iv: "edb25fc38b0b72571e3f15930406ec31"
server_key_version: "encryptionkey@0"
I20260808 02:25:39.925923 30510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7e21c0157542a482e7f630eb1ef61c"
format_stamp: "Formatted at 2026-08-08 02:25:39 on dist-test-slave-pgkr"
server_key: "2b9bee54ef37ef572ff09e9a752c010a"
server_key_iv: "edb25fc38b0b72571e3f15930406ec31"
server_key_version: "encryptionkey@0"
I20260808 02:25:39.931269 30510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:25:39.934885 30525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:39.936224 30510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:25:39.936410 30510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "5c7e21c0157542a482e7f630eb1ef61c"
format_stamp: "Formatted at 2026-08-08 02:25:39 on dist-test-slave-pgkr"
server_key: "2b9bee54ef37ef572ff09e9a752c010a"
server_key_iv: "edb25fc38b0b72571e3f15930406ec31"
server_key_version: "encryptionkey@0"
I20260808 02:25:39.936683 30510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:39.958534 30510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:39.959448 30510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:39.959772 30510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:39.961678 30510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:39.963560 30510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:39.963650 30510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:39.963732 30510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:39.963785 30510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:40.000178 30510 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:42899
I20260808 02:25:40.000285 30642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:42899 every 8 connection(s)
I20260808 02:25:40.001686 30510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:40.002321 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 30510
I20260808 02:25:40.002519 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:25:40.003075 20380 external_mini_cluster.cc:1428] Setting key 01b1c47ec51dc57d05dab4b05f062b20
I20260808 02:25:40.015349 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:40.004808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:51459 (local address 127.19.231.62:35285)
0808 02:25:40.005028 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:40.005034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:40.007072 (+ 2038us) server_negotiation.cc:407] Connection header received
0808 02:25:40.008864 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:40.008869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:40.008934 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:40.009037 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:40.010149 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.010984 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.012710 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.013147 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.013257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:40.014574 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:40.014579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:40.014582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:40.014599 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:40.014700 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:40.014715 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:40.014787 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:40.014989 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 02:25:40.015100 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:25:40.016398 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:40.005423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:35285 (local address 127.19.231.3:51459)
0808 02:25:40.006887 (+ 1464us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:40.006937 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:40.008580 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:40.009311 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:40.009320 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:40.009405 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:40.009988 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:40.010005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.011165 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.011169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:40.012504 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:40.012517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.013347 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.013352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:40.013599 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:40.014290 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:25:40.014293 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:40.014331 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:25:40.014412 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:40.014801 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:40.014810 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:40.014816 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:25:40.014959 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 02:25:40.015257 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":361,"threads_started":1}
I20260808 02:25:40.018072 30643 heartbeater.cc:347] Connected to a master server at 127.19.231.62:35285
I20260808 02:25:40.018455 30643 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:40.019378 30643 heartbeater.cc:511] Master 127.19.231.62:35285 requested a full tablet report, sending...
I20260808 02:25:40.020859 30180 ts_manager.cc:194] Registered new tserver with Master: 5c7e21c0157542a482e7f630eb1ef61c (127.19.231.3:42899)
I20260808 02:25:40.021708 30180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.3:51459
I20260808 02:25:40.032482 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:25:40.039218 30376 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:40.037450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.19.231.62:35285)
0808 02:25:40.037639 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:40.037644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:40.038210 (+ 566us) server_negotiation.cc:407] Connection header received
0808 02:25:40.038339 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:40.038345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:40.038424 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:40.038530 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:40.039038 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55896: BlockingRecv error: recv got EOF from 127.0.0.1:55896 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:25:40.039959 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 30240
I20260808 02:25:40.047966 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:40.040913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.19.231.1:40721)
0808 02:25:40.041466 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:40.041473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:40.041498 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:25:40.041551 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:40.041556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:40.041741 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:40.041857 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:40.042220 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.043295 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.044616 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.045174 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.045271 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:40.046383 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:40.046391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:40.046403 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:40.047054 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:40.047105 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:40.047127 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:40.047229 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:40.047469 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 02:25:40.047652 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
I20260808 02:25:40.049289 30369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:40.134725 30374 heartbeater.cc:503] Master 127.19.231.62:35285 was elected leader, sending a full tablet report...
I20260808 02:25:40.136168 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 30240
W20260808 02:25:40.156167 30167 connection.cc:535] server connection from 127.19.231.1:56745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:25:40.156551 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 30378
I20260808 02:25:40.164819 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:40.157439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.19.231.2:35677)
0808 02:25:40.158011 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:40.158019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:40.158050 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:25:40.158118 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:40.158124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:40.158357 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:40.158568 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:40.159017 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.160034 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.161298 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.161897 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.162037 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:40.163304 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:40.163312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:40.163325 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:40.163956 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:40.164003 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:40.164023 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:40.164151 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:40.164311 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 02:25:40.164466 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":324,"threads_started":1}
I20260808 02:25:40.166129 30502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:40.250423 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 30378
I20260808 02:25:40.271217 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 30510
I20260808 02:25:40.280113 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:40.272411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.19.231.3:42899)
0808 02:25:40.273139 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:40.273146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:40.273179 (+ 33us) server_negotiation.cc:407] Connection header received
0808 02:25:40.273240 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:40.273247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:40.273469 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:40.273599 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:40.274100 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.275325 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.276660 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:40.277263 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:40.277365 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:40.278492 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:40.278501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:40.278516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:40.279229 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:40.279275 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:40.279295 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:40.279383 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:40.279614 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 02:25:40.279735 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":396,"threads_started":1}
I20260808 02:25:40.281486 30638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:40.365356 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 30510
I20260808 02:25:40.386204 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 30148
I20260808 02:25:40.387742 30211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:40.497004 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 30148
2026-08-08T02:25:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2555 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T02:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:40Z Disabled control of system clock
I20260808 02:25:40.586902 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43185
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:25:40.936050 30662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:40.936362 30662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:40.936514 30662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:40.944128 30662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:40.944242 30662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:40.944331 30662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:40.944437 30662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:40.944506 30662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:40.954154 30662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:40.955701 30662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:40.957670 30662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:40.970435 30667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:40.970404 30670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:40.970351 30668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:40.971243 30662 server_base.cc:1034] running on GCE node
I20260808 02:25:40.972048 30662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:40.973456 30662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:40.974723 30662 hybrid_clock.cc:648] HybridClock initialized: now 1786155940974664 us; error 59 us; skew 500 ppm
I20260808 02:25:40.975090 30662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:40.977411 30662 webserver.cc:460] Webserver started at http://127.19.231.62:43287/ using document root <none> and password file <none>
I20260808 02:25:40.978137 30662 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:40.978266 30662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:40.978643 30662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:40.983827 30662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "ed956a547e964355a3cdefc20149d414"
format_stamp: "Formatted at 2026-08-08 02:25:40 on dist-test-slave-pgkr"
server_key: "223a451b1bdfb3b4f676952e75842ce6"
server_key_iv: "2f1f95fc4687b56715058055e3c1d835"
server_key_version: "encryptionkey@0"
I20260808 02:25:40.984704 30662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "ed956a547e964355a3cdefc20149d414"
format_stamp: "Formatted at 2026-08-08 02:25:40 on dist-test-slave-pgkr"
server_key: "223a451b1bdfb3b4f676952e75842ce6"
server_key_iv: "2f1f95fc4687b56715058055e3c1d835"
server_key_version: "encryptionkey@0"
I20260808 02:25:40.990150 30662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:25:40.993793 30677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:40.995292 30662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:25:40.995481 30662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "ed956a547e964355a3cdefc20149d414"
format_stamp: "Formatted at 2026-08-08 02:25:40 on dist-test-slave-pgkr"
server_key: "223a451b1bdfb3b4f676952e75842ce6"
server_key_iv: "2f1f95fc4687b56715058055e3c1d835"
server_key_version: "encryptionkey@0"
I20260808 02:25:40.995759 30662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:41.001628 30662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:41.002399 30662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:41.002755 30662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:41.021950 30662 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43185
I20260808 02:25:41.021982 30731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43185 every 8 connection(s)
I20260808 02:25:41.023442 30662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:41.024019 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 30662
I20260808 02:25:41.024276 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:25:41.025148 20380 external_mini_cluster.cc:1428] Setting key 08106f3131f5999edc5cbf045fae06cc
I20260808 02:25:41.028214 30732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:41.033900 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:41.027394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.19.231.62:43185)
0808 02:25:41.028532 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:41.028546 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:41.028589 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:25:41.030218 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:41.030239 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:41.030707 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:41.031061 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:41.032183 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:41.032189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:41.032201 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:41.032215 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:41.032328 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:41.032345 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:41.032519 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:41.032620 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 02:25:41.032906 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":439,"threads_started":1}
I20260808 02:25:41.039408 30732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414: Bootstrap starting.
I20260808 02:25:41.042302 30732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:41.043367 30732 log.cc:824] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:41.045931 30732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414: No bootstrap required, opened a new log
I20260808 02:25:41.052994 30732 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed956a547e964355a3cdefc20149d414" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43185 } }
I20260808 02:25:41.053354 30732 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:41.053457 30732 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed956a547e964355a3cdefc20149d414, State: Initialized, Role: FOLLOWER
I20260808 02:25:41.054008 30732 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed956a547e964355a3cdefc20149d414" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43185 } }
I20260808 02:25:41.054243 30732 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:41.054374 30732 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:41.054553 30732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:41.056866 30732 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed956a547e964355a3cdefc20149d414" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43185 } }
I20260808 02:25:41.057359 30732 leader_election.cc:302] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed956a547e964355a3cdefc20149d414; no voters:
I20260808 02:25:41.057953 30732 leader_election.cc:288] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:41.058106 30737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:41.058849 30737 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [term 1 LEADER]: Becoming Leader. State: Replica: ed956a547e964355a3cdefc20149d414, State: Running, Role: LEADER
I20260808 02:25:41.059468 30737 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed956a547e964355a3cdefc20149d414" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43185 } }
I20260808 02:25:41.060050 30732 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:41.062180 30738 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed956a547e964355a3cdefc20149d414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed956a547e964355a3cdefc20149d414" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43185 } } }
I20260808 02:25:41.062508 30738 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:41.063050 30739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed956a547e964355a3cdefc20149d414. Latest consensus state: current_term: 1 leader_uuid: "ed956a547e964355a3cdefc20149d414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed956a547e964355a3cdefc20149d414" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43185 } } }
I20260808 02:25:41.063391 30739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:41.063571 30744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:41.069495 30744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:41.078063 30744 catalog_manager.cc:1347] Generated new cluster ID: 8ffc4d256c8b45a1b4f5f7540ab79989
I20260808 02:25:41.078187 30744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:41.096225 30744 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:41.096705 30744 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:41.120910 30744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed956a547e964355a3cdefc20149d414: Generated new TSK 0
I20260808 02:25:41.122045 30744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:25:41.133883 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43185
--builtin_ntp_servers=127.19.231.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 02:25:41.486258 30756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:41.486573 30756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:41.486760 30756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:41.494138 30756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:41.494297 30756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:41.494493 30756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:25:41.504617 30756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:41.506091 30756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:41.508287 30756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:41.521706 30764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:41.521818 30761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:41.521776 30762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:41.522863 30756 server_base.cc:1034] running on GCE node
I20260808 02:25:41.523439 30756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:41.524431 30756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:41.525656 30756 hybrid_clock.cc:648] HybridClock initialized: now 1786155941525625 us; error 74 us; skew 500 ppm
I20260808 02:25:41.525974 30756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:41.528288 30756 webserver.cc:460] Webserver started at http://127.19.231.1:38855/ using document root <none> and password file <none>
I20260808 02:25:41.528991 30756 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:41.529109 30756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:41.529469 30756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:41.532426 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "5145d601331244d1bc95f61a1ed85343"
format_stamp: "Formatted at 2026-08-08 02:25:41 on dist-test-slave-pgkr"
server_key: "e496e26f34ae8b39e16a0867565769df"
server_key_iv: "32a5e264dffd6dcee3f5bf0abf26492c"
server_key_version: "encryptionkey@0"
I20260808 02:25:41.533234 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "5145d601331244d1bc95f61a1ed85343"
format_stamp: "Formatted at 2026-08-08 02:25:41 on dist-test-slave-pgkr"
server_key: "e496e26f34ae8b39e16a0867565769df"
server_key_iv: "32a5e264dffd6dcee3f5bf0abf26492c"
server_key_version: "encryptionkey@0"
I20260808 02:25:41.538375 30756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:25:41.541796 30770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:41.543154 30756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:25:41.543326 30756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "5145d601331244d1bc95f61a1ed85343"
format_stamp: "Formatted at 2026-08-08 02:25:41 on dist-test-slave-pgkr"
server_key: "e496e26f34ae8b39e16a0867565769df"
server_key_iv: "32a5e264dffd6dcee3f5bf0abf26492c"
server_key_version: "encryptionkey@0"
I20260808 02:25:41.543538 30756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:41.548943 30756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:41.549696 30756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:41.549968 30756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:41.550956 30756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:41.552346 30756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:41.552452 30756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:41.552579 30756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:41.552685 30756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:41.586019 30756 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:41063
I20260808 02:25:41.586117 30886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:41063 every 8 connection(s)
I20260808 02:25:41.587428 30756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:41.590930 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 30756
I20260808 02:25:41.591174 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:25:41.592023 20380 external_mini_cluster.cc:1428] Setting key cebcc8451e84a113cb40224d7c7d43f5
I20260808 02:25:41.594825 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43185
--builtin_ntp_servers=127.19.231.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:25:41.596624 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:41.590257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:50783 (local address 127.19.231.62:43185)
0808 02:25:41.590773 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:41.590777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:41.592198 (+ 1421us) server_negotiation.cc:407] Connection header received
0808 02:25:41.594181 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:41.594187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:41.594261 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:41.594381 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:41.595842 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:41.595848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:41.595853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:41.595872 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:41.595922 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:41.595934 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:41.596072 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:41.596255 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 02:25:41.596325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
I20260808 02:25:41.597559 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:41.590973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43185 (local address 127.19.231.1:50783)
0808 02:25:41.592015 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:41.592057 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:41.593918 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:41.594698 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:41.594707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:41.594802 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:41.595543 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:25:41.595545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:41.595588 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:25:41.595673 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:41.596029 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:41.596043 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:41.596050 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:25:41.596346 (+ 296us) client_negotiation.cc:240] Negotiation successful
0808 02:25:41.596625 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":329,"threads_started":1}
I20260808 02:25:41.599133 30887 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43185
I20260808 02:25:41.599442 30887 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:41.600131 30887 heartbeater.cc:511] Master 127.19.231.62:43185 requested a full tablet report, sending...
I20260808 02:25:41.602495 30694 ts_manager.cc:194] Registered new tserver with Master: 5145d601331244d1bc95f61a1ed85343 (127.19.231.1:41063)
W20260808 02:25:41.944849 30891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:41.945178 30891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:41.945333 30891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:41.952720 30891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:41.952862 30891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:41.953038 30891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:25:41.962276 30891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:41.963644 30891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:41.965507 30891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:41.979817 30896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:41.979924 30899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:41.979947 30897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:41.981398 30891 server_base.cc:1034] running on GCE node
I20260808 02:25:41.982009 30891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:41.983003 30891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:41.984221 30891 hybrid_clock.cc:648] HybridClock initialized: now 1786155941984176 us; error 38 us; skew 500 ppm
I20260808 02:25:41.984539 30891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:41.986907 30891 webserver.cc:460] Webserver started at http://127.19.231.2:33911/ using document root <none> and password file <none>
I20260808 02:25:41.987644 30891 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:41.987782 30891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:41.988107 30891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:41.991122 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "bea29e3cd584446290c2240627ed45a8"
format_stamp: "Formatted at 2026-08-08 02:25:41 on dist-test-slave-pgkr"
server_key: "8b26c90deea8b342a3aa74fc2dee44a8"
server_key_iv: "0b5fb62913529551dc53beb23e7eb280"
server_key_version: "encryptionkey@0"
I20260808 02:25:41.991922 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "bea29e3cd584446290c2240627ed45a8"
format_stamp: "Formatted at 2026-08-08 02:25:41 on dist-test-slave-pgkr"
server_key: "8b26c90deea8b342a3aa74fc2dee44a8"
server_key_iv: "0b5fb62913529551dc53beb23e7eb280"
server_key_version: "encryptionkey@0"
I20260808 02:25:41.997057 30891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:25:42.000525 30905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:42.001860 30891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:25:42.002035 30891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "bea29e3cd584446290c2240627ed45a8"
format_stamp: "Formatted at 2026-08-08 02:25:41 on dist-test-slave-pgkr"
server_key: "8b26c90deea8b342a3aa74fc2dee44a8"
server_key_iv: "0b5fb62913529551dc53beb23e7eb280"
server_key_version: "encryptionkey@0"
I20260808 02:25:42.002261 30891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:42.008015 30891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:42.008842 30891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:42.009122 30891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:42.010038 30891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:42.011533 30891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:42.011653 30891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:42.011790 30891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:42.011881 30891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:42.045954 30891 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:34751
I20260808 02:25:42.046006 31022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:34751 every 8 connection(s)
I20260808 02:25:42.047405 30891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:42.054193 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 30891
I20260808 02:25:42.054417 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:25:42.055073 20380 external_mini_cluster.cc:1428] Setting key a10ce327c482996889805ed607c46e82
I20260808 02:25:42.055945 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:42.050047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:56707 (local address 127.19.231.62:43185)
0808 02:25:42.050248 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:42.050253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:42.051704 (+ 1451us) server_negotiation.cc:407] Connection header received
0808 02:25:42.053477 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:42.053483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:42.053573 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:42.053685 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:42.055116 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:42.055121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:42.055126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:42.055156 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:42.055213 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:42.055229 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:42.055353 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:42.055544 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 02:25:42.055618 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:25:42.056983 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:42.050588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43185 (local address 127.19.231.2:56707)
0808 02:25:42.051526 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:42.051570 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:42.053171 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:42.053937 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:42.053946 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:42.054040 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:42.054762 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:25:42.054765 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:42.054827 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:25:42.054950 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:42.055324 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:42.055335 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:42.055343 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:25:42.055631 (+ 288us) client_negotiation.cc:240] Negotiation successful
0808 02:25:42.055907 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":262,"threads_started":1}
I20260808 02:25:42.057956 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43185
--builtin_ntp_servers=127.19.231.20:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 02:25:42.059937 31023 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43185
I20260808 02:25:42.060236 31023 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:42.060863 31023 heartbeater.cc:511] Master 127.19.231.62:43185 requested a full tablet report, sending...
I20260808 02:25:42.062269 30694 ts_manager.cc:194] Registered new tserver with Master: bea29e3cd584446290c2240627ed45a8 (127.19.231.2:34751)
W20260808 02:25:42.398759 31026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:42.399044 31026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:42.399132 31026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:42.406456 31026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:42.406551 31026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:42.406678 31026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:25:42.415436 31026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:42.416762 31026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:42.418509 31026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:42.431480 31031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:42.431538 31032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:42.431567 31034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:42.432468 31026 server_base.cc:1034] running on GCE node
I20260808 02:25:42.433127 31026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:42.434113 31026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:42.435359 31026 hybrid_clock.cc:648] HybridClock initialized: now 1786155942435309 us; error 45 us; skew 500 ppm
I20260808 02:25:42.435700 31026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:25:42.442359 31026 webserver.cc:460] Webserver started at http://127.19.231.3:37571/ using document root <none> and password file <none>
I20260808 02:25:42.443259 31026 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:42.443403 31026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:42.443763 31026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:42.446970 31026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "68bfbcea82234368ae32550e24354c95"
format_stamp: "Formatted at 2026-08-08 02:25:42 on dist-test-slave-pgkr"
server_key: "d9b317e9325182f4b10300c8a0e37e40"
server_key_iv: "00429e5c243658358e1718381fdc58e3"
server_key_version: "encryptionkey@0"
I20260808 02:25:42.447894 31026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "68bfbcea82234368ae32550e24354c95"
format_stamp: "Formatted at 2026-08-08 02:25:42 on dist-test-slave-pgkr"
server_key: "d9b317e9325182f4b10300c8a0e37e40"
server_key_iv: "00429e5c243658358e1718381fdc58e3"
server_key_version: "encryptionkey@0"
I20260808 02:25:42.453401 31026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:25:42.457043 31040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:42.458455 31026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:25:42.458716 31026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "68bfbcea82234368ae32550e24354c95"
format_stamp: "Formatted at 2026-08-08 02:25:42 on dist-test-slave-pgkr"
server_key: "d9b317e9325182f4b10300c8a0e37e40"
server_key_iv: "00429e5c243658358e1718381fdc58e3"
server_key_version: "encryptionkey@0"
I20260808 02:25:42.458935 31026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:42.464625 31026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:42.465402 31026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:42.465669 31026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:42.466727 31026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:42.468117 31026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:42.468238 31026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:42.468366 31026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:42.468456 31026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:42.501771 31026 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:37075
I20260808 02:25:42.501817 31154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:37075 every 8 connection(s)
I20260808 02:25:42.503361 31026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:42.507680 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 31026
I20260808 02:25:42.507912 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:25:42.508512 20380 external_mini_cluster.cc:1428] Setting key f3993dc3187ba8de9b292ae28ac9546a
I20260808 02:25:42.512740 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:42.506218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:40417 (local address 127.19.231.62:43185)
0808 02:25:42.506421 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:42.506426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:42.508199 (+ 1773us) server_negotiation.cc:407] Connection header received
0808 02:25:42.510251 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:42.510257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:42.510334 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:42.510442 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:42.512059 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:42.512064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:42.512067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:42.512082 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:42.512113 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:42.512124 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:42.512224 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:42.512473 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 02:25:42.512523 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:25:42.513893 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:42.506962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43185 (local address 127.19.231.3:40417)
0808 02:25:42.507990 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:42.508074 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:42.510000 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:42.510736 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:42.510749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:42.510876 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:42.511755 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:25:42.511759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:42.511814 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:25:42.511913 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:42.512250 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:42.512273 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:42.512280 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:25:42.512542 (+ 262us) client_negotiation.cc:240] Negotiation successful
0808 02:25:42.512745 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":322,"threads_started":1}
I20260808 02:25:42.515342 31155 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43185
I20260808 02:25:42.515635 31155 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:42.516220 31155 heartbeater.cc:511] Master 127.19.231.62:43185 requested a full tablet report, sending...
I20260808 02:25:42.517652 30694 ts_manager.cc:194] Registered new tserver with Master: 68bfbcea82234368ae32550e24354c95 (127.19.231.3:37075)
I20260808 02:25:42.523974 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 02:25:42.530822 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:25:42.529135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.19.231.62:43185)
0808 02:25:42.529297 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:42.529301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:42.529944 (+ 643us) server_negotiation.cc:407] Connection header received
0808 02:25:42.530058 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:42.530062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:42.530117 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:42.530203 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:42.530542 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55532: BlockingRecv error: recv got EOF from 127.0.0.1:55532 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:25:42.531849 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 30756
I20260808 02:25:42.535369 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:42.532774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.19.231.1:41063)
0808 02:25:42.533340 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:42.533347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:42.533372 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:25:42.533427 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:42.533433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:42.533637 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:42.533757 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:42.534760 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:42.534767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:42.534779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:42.534803 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:42.534852 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:42.534869 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:42.534973 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:42.535066 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 02:25:42.535139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":298,"threads_started":1}
I20260808 02:25:42.536536 30882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:42.635174 30887 heartbeater.cc:503] Master 127.19.231.62:43185 was elected leader, sending a full tablet report...
I20260808 02:25:42.636965 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 30756
W20260808 02:25:42.656613 30679 connection.cc:535] server connection from 127.19.231.1:50783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:25:42.657008 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 30891
I20260808 02:25:42.660696 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:42.658022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.19.231.2:34751)
0808 02:25:42.658596 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:42.658603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:42.658689 (+ 86us) server_negotiation.cc:407] Connection header received
0808 02:25:42.658765 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:42.658771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:42.658999 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:42.659134 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:42.660040 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:42.660047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:42.660058 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:42.660071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:42.660118 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:42.660136 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:42.660301 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:42.660363 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 02:25:42.660454 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
I20260808 02:25:42.662035 31018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:42.744712 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 30891
I20260808 02:25:42.765254 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 31026
I20260808 02:25:42.769692 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:42.766420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.19.231.3:37075)
0808 02:25:42.767263 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:42.767270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:42.767301 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:25:42.767364 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:42.767371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:42.767584 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:42.767756 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:42.768920 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:42.768926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:42.768938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:25:42.768955 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:42.768995 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:25:42.769008 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:42.769204 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:42.769283 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 02:25:42.769388 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":492,"threads_started":1}
I20260808 02:25:42.771025 31150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:42.853188 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 31026
I20260808 02:25:42.873553 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 30662
I20260808 02:25:42.874869 30727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:42.978343 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 30662
2026-08-08T02:25:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2481 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:43 dist-test-slave-pgkr krb5kdc[31174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:43 dist-test-slave-pgkr krb5kdc[31174](info): set up 2 sockets
Aug 08 02:25:43 dist-test-slave-pgkr krb5kdc[31174](info): commencing operation
krb5kdc: starting...
W20260808 02:25:44.148535 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:25:44 dist-test-slave-pgkr krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:25:44.446344 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43481
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:44.801532 31188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:44.801815 31188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:44.801978 31188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:44.809749 31188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:25:44.809856 31188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:44.809957 31188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:44.810063 31188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:25:44.810132 31188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:25:44.820344 31188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:44.821743 31188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:44.823120 31188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:44.835318 31193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:44.835239 31194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:44.835294 31196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:44.836012 31188 server_base.cc:1034] running on GCE node
I20260808 02:25:44.836740 31188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:44.838124 31188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:44.839332 31188 hybrid_clock.cc:648] HybridClock initialized: now 1786155944839286 us; error 33 us; skew 500 ppm
I20260808 02:25:44.839705 31188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:44 dist-test-slave-pgkr krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155944, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:44.847383 31188 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:25:44.849126 31188 webserver.cc:460] Webserver started at http://127.19.231.62:40055/ using document root <none> and password file <none>
I20260808 02:25:44.849893 31188 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:44.850070 31188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:44.850422 31188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:44.853770 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "2b3363f78c504190a1128ebe782ddfb4"
format_stamp: "Formatted at 2026-08-08 02:25:44 on dist-test-slave-pgkr"
server_key: "7c3cd66afaca8cc469f197bcf8b07fc0"
server_key_iv: "8c0e0b4dff68dd88fe4ab5156ffe9df6"
server_key_version: "encryptionkey@0"
I20260808 02:25:44.854761 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "2b3363f78c504190a1128ebe782ddfb4"
format_stamp: "Formatted at 2026-08-08 02:25:44 on dist-test-slave-pgkr"
server_key: "7c3cd66afaca8cc469f197bcf8b07fc0"
server_key_iv: "8c0e0b4dff68dd88fe4ab5156ffe9df6"
server_key_version: "encryptionkey@0"
I20260808 02:25:44.860451 31188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:25:44.864450 31203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:44.865834 31188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:25:44.866016 31188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "2b3363f78c504190a1128ebe782ddfb4"
format_stamp: "Formatted at 2026-08-08 02:25:44 on dist-test-slave-pgkr"
server_key: "7c3cd66afaca8cc469f197bcf8b07fc0"
server_key_iv: "8c0e0b4dff68dd88fe4ab5156ffe9df6"
server_key_version: "encryptionkey@0"
I20260808 02:25:44.866221 31188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:44.883052 31188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:44.887548 31188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:44.887846 31188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:44.907321 31188 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43481
I20260808 02:25:44.907356 31254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43481 every 8 connection(s)
I20260808 02:25:44.908721 31188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:25:44.912514 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:44.916124 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 31188
I20260808 02:25:44.916397 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:25:44.917105 20380 external_mini_cluster.cc:1428] Setting key 5616fc40d0e0a6ee43dbbd96d29a55ea
I20260808 02:25:44.925138 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4: Bootstrap starting.
I20260808 02:25:44.928963 31255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:44.930239 31255 log.cc:824] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:44.933544 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4: No bootstrap required, opened a new log
Aug 08 02:25:44 dist-test-slave-pgkr krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:44.943192 31255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3363f78c504190a1128ebe782ddfb4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43481 } }
I20260808 02:25:44.943550 31255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:44.943657 31255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3363f78c504190a1128ebe782ddfb4, State: Initialized, Role: FOLLOWER
I20260808 02:25:44.944137 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:44.919499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.19.231.62:43481)
0808 02:25:44.920354 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:44.920370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:44.920421 (+ 51us) server_negotiation.cc:407] Connection header received
0808 02:25:44.923344 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:44.923374 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:44.923890 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:44.924334 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:44.926108 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:44.928195 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:44.929686 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:44.930850 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:44.931088 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:44.935846 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:44.935858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:44.935872 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:44.935912 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:44.939564 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:44.940404 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:44.940414 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:44.940421 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:44.940644 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:44.941079 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:44.941083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:44.941085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:44.941666 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:44.941908 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:44.942414 (+ 506us) server_negotiation.cc:299] Negotiation successful
0808 02:25:44.942896 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":349,"threads_started":1}
I20260808 02:25:44.944340 31255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3363f78c504190a1128ebe782ddfb4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43481 } }
I20260808 02:25:44.944619 31255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:25:44.944804 31255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:25:44.944986 31255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:44.947494 31255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3363f78c504190a1128ebe782ddfb4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43481 } }
I20260808 02:25:44.948007 31255 leader_election.cc:302] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b3363f78c504190a1128ebe782ddfb4; no voters:
I20260808 02:25:44.948639 31255 leader_election.cc:288] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:25:44.948881 31260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:44.949666 31260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3363f78c504190a1128ebe782ddfb4, State: Running, Role: LEADER
I20260808 02:25:44.950253 31260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3363f78c504190a1128ebe782ddfb4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43481 } }
I20260808 02:25:44.950752 31255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:25:44.953099 31262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b3363f78c504190a1128ebe782ddfb4. Latest consensus state: current_term: 1 leader_uuid: "2b3363f78c504190a1128ebe782ddfb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3363f78c504190a1128ebe782ddfb4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43481 } } }
I20260808 02:25:44.953490 31262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:44.954177 31261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b3363f78c504190a1128ebe782ddfb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3363f78c504190a1128ebe782ddfb4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43481 } } }
I20260808 02:25:44.954667 31265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:25:44.955299 31261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:25:44.961128 31265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:25:44.970908 31265 catalog_manager.cc:1347] Generated new cluster ID: 4a02b8c6266b4526affcad046131eeb1
I20260808 02:25:44.971066 31265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:25:44.991057 31265 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:25:44.992317 31265 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:25:45.007699 31265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4: Generated new TSK 0
I20260808 02:25:45.008718 31265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:25:45.091316 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43481
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:45.439476 31281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:45.439808 31281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:45.439970 31281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:45.447566 31281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:45.447736 31281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:45.447906 31281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:25:45.458181 31281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:45.459612 31281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:45.460991 31281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:45.474540 31287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:45.474735 31289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:45.474766 31286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:45.475755 31281 server_base.cc:1034] running on GCE node
I20260808 02:25:45.476428 31281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:45.477381 31281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:45.478673 31281 hybrid_clock.cc:648] HybridClock initialized: now 1786155945478585 us; error 60 us; skew 500 ppm
I20260808 02:25:45.478999 31281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:45 dist-test-slave-pgkr krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155945, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:45.482366 31281 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:25:45.483947 31281 webserver.cc:460] Webserver started at http://127.19.231.1:45267/ using document root <none> and password file <none>
I20260808 02:25:45.484625 31281 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:45.484725 31281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:45.484977 31281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:45.488003 31281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "1d0ae2a95c06457589baea3b342d922d"
format_stamp: "Formatted at 2026-08-08 02:25:45 on dist-test-slave-pgkr"
server_key: "a981386388b839083cc9e2e570f8ec29"
server_key_iv: "585bdf6e412a58e7e00dde5ddde1aeb1"
server_key_version: "encryptionkey@0"
I20260808 02:25:45.488766 31281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "1d0ae2a95c06457589baea3b342d922d"
format_stamp: "Formatted at 2026-08-08 02:25:45 on dist-test-slave-pgkr"
server_key: "a981386388b839083cc9e2e570f8ec29"
server_key_iv: "585bdf6e412a58e7e00dde5ddde1aeb1"
server_key_version: "encryptionkey@0"
I20260808 02:25:45.494004 31281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 02:25:45.497596 31296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:45.499140 31281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 02:25:45.499300 31281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "1d0ae2a95c06457589baea3b342d922d"
format_stamp: "Formatted at 2026-08-08 02:25:45 on dist-test-slave-pgkr"
server_key: "a981386388b839083cc9e2e570f8ec29"
server_key_iv: "585bdf6e412a58e7e00dde5ddde1aeb1"
server_key_version: "encryptionkey@0"
I20260808 02:25:45.499490 31281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:45.519330 31281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:45.523941 31281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:45.524262 31281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:45.525306 31281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:45.526759 31281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:45.526837 31281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:45.526983 31281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:45.527036 31281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:45.564697 31281 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:36701
I20260808 02:25:45.564743 31415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:36701 every 8 connection(s)
I20260808 02:25:45.566102 31281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:25:45.572430 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 31281
I20260808 02:25:45.572628 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:25:45.573267 20380 external_mini_cluster.cc:1428] Setting key 83ab1249a292132216e3c8cf5ad2c603
Aug 08 02:25:45 dist-test-slave-pgkr krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155945, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:45.587502 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:45.568882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:48763 (local address 127.19.231.62:43481)
0808 02:25:45.569496 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:45.569501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:45.570556 (+ 1055us) server_negotiation.cc:407] Connection header received
0808 02:25:45.572746 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:45.572752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:45.572906 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:45.573054 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:45.574659 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:45.575378 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:45.576926 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:45.577339 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:45.577469 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:45.580955 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:45.580960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:45.580963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:45.581000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:45.584200 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:45.585080 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:45.585085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:45.585087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:45.585211 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:45.585644 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:45.585651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:45.585654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:45.585927 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:45.586084 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:45.586840 (+ 756us) server_negotiation.cc:299] Negotiation successful
0808 02:25:45.587072 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":245,"threads_started":1}
I20260808 02:25:45.588585 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:45.569554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43481 (local address 127.19.231.1:48763)
0808 02:25:45.570381 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:45.570442 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:45.572380 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:45.573332 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:45.573340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:45.573895 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:45.574469 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:45.574487 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:45.575528 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:45.575534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:45.576778 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:45.576791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:45.577579 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:45.577586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:45.577839 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:45.578521 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:45.578550 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:45.580745 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:45.584358 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:45.584367 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:45.584383 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:45.584933 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:45.585323 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:45.585328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:45.585332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:45.585528 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:45.586074 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:45.586082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:45.586422 (+ 340us) client_negotiation.cc:769] Sending connection context
0808 02:25:45.586741 (+ 319us) client_negotiation.cc:240] Negotiation successful
0808 02:25:45.587315 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":309,"threads_started":1}
I20260808 02:25:45.590255 31416 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43481
I20260808 02:25:45.590577 31416 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:45.591398 31416 heartbeater.cc:511] Master 127.19.231.62:43481 requested a full tablet report, sending...
I20260808 02:25:45.593611 31220 ts_manager.cc:194] Registered new tserver with Master: 1d0ae2a95c06457589baea3b342d922d (127.19.231.1:36701)
I20260808 02:25:45.595160 31220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:48763
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:25:45.639726 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43481
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:46.003831 31422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:46.004139 31422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:46.004281 31422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:46.011844 31422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:46.011998 31422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:46.012202 31422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:25:46.022109 31422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:46.023563 31422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:46.025009 31422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:46.038851 31430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:46.038936 31427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:46.038882 31428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:46.039794 31422 server_base.cc:1034] running on GCE node
I20260808 02:25:46.040401 31422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:46.041533 31422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:46.042768 31422 hybrid_clock.cc:648] HybridClock initialized: now 1786155946042730 us; error 51 us; skew 500 ppm
I20260808 02:25:46.043093 31422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:46 dist-test-slave-pgkr krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:46.046262 31422 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:25:46.047659 31422 webserver.cc:460] Webserver started at http://127.19.231.2:44071/ using document root <none> and password file <none>
I20260808 02:25:46.048365 31422 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:46.048503 31422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:46.048803 31422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:46.051961 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "5e59eb10a8b34513bfc212912fd42edc"
format_stamp: "Formatted at 2026-08-08 02:25:46 on dist-test-slave-pgkr"
server_key: "0bf7918a383d0da51b142b85481c542f"
server_key_iv: "eb1ff84a8ec1aebc2c683b37825f6218"
server_key_version: "encryptionkey@0"
I20260808 02:25:46.052723 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "5e59eb10a8b34513bfc212912fd42edc"
format_stamp: "Formatted at 2026-08-08 02:25:46 on dist-test-slave-pgkr"
server_key: "0bf7918a383d0da51b142b85481c542f"
server_key_iv: "eb1ff84a8ec1aebc2c683b37825f6218"
server_key_version: "encryptionkey@0"
I20260808 02:25:46.057909 31422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:25:46.061383 31437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:46.062870 31422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:25:46.063053 31422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "5e59eb10a8b34513bfc212912fd42edc"
format_stamp: "Formatted at 2026-08-08 02:25:46 on dist-test-slave-pgkr"
server_key: "0bf7918a383d0da51b142b85481c542f"
server_key_iv: "eb1ff84a8ec1aebc2c683b37825f6218"
server_key_version: "encryptionkey@0"
I20260808 02:25:46.063254 31422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:46.075948 31422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:46.080088 31422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:46.080408 31422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:46.081406 31422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:46.082837 31422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:46.082935 31422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:46.083081 31422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:46.083150 31422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:46.116886 31422 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:41151
I20260808 02:25:46.116925 31552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:41151 every 8 connection(s)
I20260808 02:25:46.118222 31422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:25:46.119678 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 31422
I20260808 02:25:46.119868 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:25:46.120368 20380 external_mini_cluster.cc:1428] Setting key 21ddbba01217278f313e01af62367e05
Aug 08 02:25:46 dist-test-slave-pgkr krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:46.139802 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.121103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:39583 (local address 127.19.231.62:43481)
0808 02:25:46.121609 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.121613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.122958 (+ 1345us) server_negotiation.cc:407] Connection header received
0808 02:25:46.124746 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.124754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.124871 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.124989 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:46.126519 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.127637 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.129159 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.129877 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.130018 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.133459 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:46.133468 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:46.133475 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:46.133522 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:46.136500 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.137429 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.137433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.137435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.137562 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.138039 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.138056 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.138058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.138308 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:46.138415 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.139353 (+ 938us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.139516 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
I20260808 02:25:46.141429 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.121760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43481 (local address 127.19.231.2:39583)
0808 02:25:46.122787 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:46.122844 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:46.124487 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:46.125205 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:46.125212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:46.125763 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:46.126346 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.126364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.127774 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.127778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.129015 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.129028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.130080 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.130084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.130290 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.131004 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:46.131026 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:46.133235 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:46.136638 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:46.136658 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:46.136677 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:46.137322 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:46.137671 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:46.137678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:46.137685 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:46.137936 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:46.138440 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:46.138451 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:46.138974 (+ 523us) client_negotiation.cc:769] Sending connection context
0808 02:25:46.139292 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 02:25:46.139888 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":366,"threads_started":1}
I20260808 02:25:46.142956 31553 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43481
I20260808 02:25:46.143296 31553 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:46.144012 31553 heartbeater.cc:511] Master 127.19.231.62:43481 requested a full tablet report, sending...
I20260808 02:25:46.145527 31220 ts_manager.cc:194] Registered new tserver with Master: 5e59eb10a8b34513bfc212912fd42edc (127.19.231.2:41151)
I20260808 02:25:46.146446 31220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:39583
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:25:46.189378 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43481
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:25:46.531246 31559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:25:46.531533 31559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:25:46.531697 31559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:25:46.538880 31559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:25:46.538980 31559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:25:46.539176 31559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:25:46.548735 31559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:25:46.550033 31559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:25:46.551419 31559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:25:46.564502 31565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:25:46.564543 31568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:46.565589 31559 server_base.cc:1034] running on GCE node
W20260808 02:25:46.569851 31566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:25:46.570573 31559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:25:46.571619 31559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:25:46.572845 31559 hybrid_clock.cc:648] HybridClock initialized: now 1786155946572795 us; error 44 us; skew 500 ppm
I20260808 02:25:46.573164 31559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:25:46 dist-test-slave-pgkr krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:25:46.576346 31559 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:25:46.577793 31559 webserver.cc:460] Webserver started at http://127.19.231.3:40489/ using document root <none> and password file <none>
I20260808 02:25:46.578495 31559 fs_manager.cc:359] Metadata directory not provided
I20260808 02:25:46.578595 31559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:25:46.578999 31559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:25:46.581899 31559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "b8123a7b2127431eb49028b6879515bd"
format_stamp: "Formatted at 2026-08-08 02:25:46 on dist-test-slave-pgkr"
server_key: "387526e5577b7b0c8df9270051650e8b"
server_key_iv: "a59a1f406410b29f4875adbc3baf21a6"
server_key_version: "encryptionkey@0"
I20260808 02:25:46.582590 31559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "b8123a7b2127431eb49028b6879515bd"
format_stamp: "Formatted at 2026-08-08 02:25:46 on dist-test-slave-pgkr"
server_key: "387526e5577b7b0c8df9270051650e8b"
server_key_iv: "a59a1f406410b29f4875adbc3baf21a6"
server_key_version: "encryptionkey@0"
I20260808 02:25:46.587745 31559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 02:25:46.591243 31575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:46.592617 31559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:25:46.592785 31559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "b8123a7b2127431eb49028b6879515bd"
format_stamp: "Formatted at 2026-08-08 02:25:46 on dist-test-slave-pgkr"
server_key: "387526e5577b7b0c8df9270051650e8b"
server_key_iv: "a59a1f406410b29f4875adbc3baf21a6"
server_key_version: "encryptionkey@0"
I20260808 02:25:46.592962 31559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:25:46.598033 31416 heartbeater.cc:503] Master 127.19.231.62:43481 was elected leader, sending a full tablet report...
I20260808 02:25:46.622464 31559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:25:46.627158 31559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:25:46.627480 31559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:25:46.628494 31559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:25:46.630039 31559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:25:46.630123 31559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:46.630290 31559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:25:46.630342 31559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:25:46.665611 31559 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:44435
I20260808 02:25:46.665642 31690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:44435 every 8 connection(s)
I20260808 02:25:46.667145 31559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:25:46.669956 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 31559
I20260808 02:25:46.670205 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:25:46.670773 20380 external_mini_cluster.cc:1428] Setting key 125f0ccf7d515126a7d30d2a7b4f24a1
Aug 08 02:25:46 dist-test-slave-pgkr krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155946, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:25:46.688568 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.669974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:42873 (local address 127.19.231.62:43481)
0808 02:25:46.670514 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.670519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.671900 (+ 1381us) server_negotiation.cc:407] Connection header received
0808 02:25:46.673956 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.673962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.674065 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.674181 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:46.675910 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.676688 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.678257 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.678696 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.678789 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.682418 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:46.682423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:46.682426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:46.682458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:46.685336 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.686363 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.686366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.686368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.686528 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.687041 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.687044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.687046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.687313 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:46.687435 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.688121 (+ 686us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.688308 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":305,"threads_started":1}
I20260808 02:25:46.689635 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.670727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43481 (local address 127.19.231.3:42873)
0808 02:25:46.671759 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:46.671801 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:46.673689 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:46.674431 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:46.674440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:46.675098 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:25:46.675728 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.675748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.676841 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.676845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.678125 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.678138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.678916 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.678922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.679153 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.679806 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:25:46.679830 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:25:46.682205 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:25:46.685580 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:46.685588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:46.685601 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:46.686250 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:46.686676 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:25:46.686683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:25:46.686688 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:25:46.686919 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:25:46.687430 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:25:46.687436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:25:46.687782 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 02:25:46.688041 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 02:25:46.688466 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":411,"threads_started":1}
I20260808 02:25:46.691037 31691 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43481
I20260808 02:25:46.691318 31691 heartbeater.cc:464] Registering TS with master...
I20260808 02:25:46.691994 31691 heartbeater.cc:511] Master 127.19.231.62:43481 requested a full tablet report, sending...
I20260808 02:25:46.693373 31220 ts_manager.cc:194] Registered new tserver with Master: b8123a7b2127431eb49028b6879515bd (127.19.231.3:44435)
I20260808 02:25:46.694288 31220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:42873
I20260808 02:25:46.698925 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:25:46.716260 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.703806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.19.231.62:43481)
0808 02:25:46.703979 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.703984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.704719 (+ 735us) server_negotiation.cc:407] Connection header received
0808 02:25:46.704774 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.704777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.704839 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.704925 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:46.706441 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.707256 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.708564 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.708994 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.709108 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.711414 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:46.711419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:46.711422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:46.711453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:46.713935 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.714527 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.714531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.714573 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.714704 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.715016 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.715019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.715021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.715328 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:46.715511 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.715838 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.716018 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:25:46.720237 31220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:25:46.727176 31220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:25:46.760815 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.739254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.19.231.1:36701)
0808 02:25:46.750524 (+ 11270us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.750533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.751319 (+ 786us) server_negotiation.cc:407] Connection header received
0808 02:25:46.751523 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.751531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.751771 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.751956 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:46.752966 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.754573 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.757707 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.759283 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.759416 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.759671 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.760296 (+ 625us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.760456 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11101,"thread_start_us":11268,"threads_started":1}
I20260808 02:25:46.760839 31706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.738875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:36701 (local address 127.0.0.1:53782)
0808 02:25:46.751146 (+ 12271us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:46.751163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:46.751326 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:46.752233 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:46.752242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:46.752278 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:46.752747 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.752761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.754764 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.754769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.757478 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.757527 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.759807 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.759813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.760050 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.760106 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 02:25:46.760242 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 02:25:46.760453 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12130,"thread_start_us":11991,"threads_started":1}
I20260808 02:25:46.761250 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.738881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.19.231.2:41151)
0808 02:25:46.750048 (+ 11167us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.750063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.750881 (+ 818us) server_negotiation.cc:407] Connection header received
0808 02:25:46.751219 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.751231 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.751531 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.751735 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:46.752859 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.754574 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.758023 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.759583 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.759724 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.759898 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.760505 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.760702 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10928,"thread_start_us":10747,"threads_started":1}
I20260808 02:25:46.762128 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.738613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41151 (local address 127.0.0.1:46258)
0808 02:25:46.750642 (+ 12029us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:46.750712 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:46.750949 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:46.752036 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:46.752047 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:46.752076 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:46.752631 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.752646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.754764 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.754769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.757821 (+ 3052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.757839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.760050 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.760059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.760266 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.760306 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:25:46.761561 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0808 02:25:46.761707 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11628,"mutex_wait_us":48,"thread_start_us":11430,"threads_started":1}
I20260808 02:25:46.765144 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.751505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:44435 (local address 127.0.0.1:58792)
0808 02:25:46.755352 (+ 3847us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:46.755369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:46.755472 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:46.757573 (+ 2101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:46.757577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:46.757602 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:46.758131 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.758145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.760272 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.760277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.762526 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.762540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.764326 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.764332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.764495 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.764528 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 02:25:46.764641 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 02:25:46.764763 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3720,"mutex_wait_us":3,"thread_start_us":3537,"threads_started":1}
I20260808 02:25:46.765190 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.751530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.19.231.3:44435)
0808 02:25:46.757049 (+ 5519us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.757055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.757082 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:25:46.757156 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.757161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.757340 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.757539 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:46.758354 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.760112 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.762728 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.764105 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.764250 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.764469 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.764681 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.764843 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5370,"thread_start_us":5228,"threads_started":1}
I20260808 02:25:46.765069 31348 tablet_service.cc:1467] Processing CreateTablet for tablet 4159a525a3024e24ab916c7d80c0eb50 (DEFAULT_TABLE table=test-table [id=f97002f775df4e0ca45d4e744244bd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:46.765782 31486 tablet_service.cc:1467] Processing CreateTablet for tablet 4159a525a3024e24ab916c7d80c0eb50 (DEFAULT_TABLE table=test-table [id=f97002f775df4e0ca45d4e744244bd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:46.767437 31486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4159a525a3024e24ab916c7d80c0eb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:46.768451 31623 tablet_service.cc:1467] Processing CreateTablet for tablet 4159a525a3024e24ab916c7d80c0eb50 (DEFAULT_TABLE table=test-table [id=f97002f775df4e0ca45d4e744244bd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:25:46.769649 31348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4159a525a3024e24ab916c7d80c0eb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:46.769896 31623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4159a525a3024e24ab916c7d80c0eb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:25:46.782667 31710 tablet_bootstrap.cc:492] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Bootstrap starting.
I20260808 02:25:46.784801 31711 tablet_bootstrap.cc:492] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Bootstrap starting.
I20260808 02:25:46.786568 31712 tablet_bootstrap.cc:492] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Bootstrap starting.
I20260808 02:25:46.786765 31710 tablet_bootstrap.cc:654] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:46.788549 31711 tablet_bootstrap.cc:654] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:46.788518 31710 log.cc:824] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:46.789742 31712 tablet_bootstrap.cc:654] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Neither blocks nor log segments found. Creating new log.
I20260808 02:25:46.790120 31711 log.cc:824] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:46.790822 31712 log.cc:824] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:25:46.793324 31711 tablet_bootstrap.cc:492] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: No bootstrap required, opened a new log
I20260808 02:25:46.793586 31711 ts_tablet_manager.cc:1397] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 02:25:46.793898 31712 tablet_bootstrap.cc:492] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: No bootstrap required, opened a new log
I20260808 02:25:46.794189 31712 ts_tablet_manager.cc:1397] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 02:25:46.794224 31710 tablet_bootstrap.cc:492] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: No bootstrap required, opened a new log
I20260808 02:25:46.794471 31710 ts_tablet_manager.cc:1397] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260808 02:25:46.801810 31711 raft_consensus.cc:348] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.802183 31711 raft_consensus.cc:374] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:46.802326 31711 raft_consensus.cc:729] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e59eb10a8b34513bfc212912fd42edc, State: Initialized, Role: FOLLOWER
I20260808 02:25:46.802965 31711 consensus_queue.cc:260] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.805028 31712 raft_consensus.cc:348] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.805496 31712 raft_consensus.cc:374] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:46.805276 31710 raft_consensus.cc:348] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.805598 31712 raft_consensus.cc:729] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0ae2a95c06457589baea3b342d922d, State: Initialized, Role: FOLLOWER
I20260808 02:25:46.805737 31710 raft_consensus.cc:374] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:25:46.805859 31710 raft_consensus.cc:729] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8123a7b2127431eb49028b6879515bd, State: Initialized, Role: FOLLOWER
I20260808 02:25:46.806270 31712 consensus_queue.cc:260] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.806499 31710 consensus_queue.cc:260] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.808157 31711 ts_tablet_manager.cc:1428] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260808 02:25:46.808670 31553 heartbeater.cc:503] Master 127.19.231.62:43481 was elected leader, sending a full tablet report...
I20260808 02:25:46.810982 31719 raft_consensus.cc:482] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:25:46.812239 31719 raft_consensus.cc:504] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.815398 31719 leader_election.cc:288] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e59eb10a8b34513bfc212912fd42edc (127.19.231.2:41151), b8123a7b2127431eb49028b6879515bd (127.19.231.3:44435)
I20260808 02:25:46.819198 31691 heartbeater.cc:503] Master 127.19.231.62:43481 was elected leader, sending a full tablet report...
I20260808 02:25:46.819857 31710 ts_tablet_manager.cc:1428] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Time spent starting tablet: real 0.025s user 0.015s sys 0.011s
I20260808 02:25:46.818434 31712 ts_tablet_manager.cc:1428] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Time spent starting tablet: real 0.024s user 0.012s sys 0.009s
W20260808 02:25:46.818496 31417 tablet.cc:2369] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:46.828388 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.816232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:36215 (local address 127.19.231.3:44435)
0808 02:25:46.816769 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.816779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.821565 (+ 4786us) server_negotiation.cc:407] Connection header received
0808 02:25:46.821923 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.821929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.822020 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.822118 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:46.823238 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.824474 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.826532 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.827379 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.827564 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.827659 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.827981 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.828117 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
I20260808 02:25:46.828400 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.816485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:37629 (local address 127.19.231.2:41151)
0808 02:25:46.816972 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.816980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.822315 (+ 5335us) server_negotiation.cc:407] Connection header received
0808 02:25:46.822479 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.822484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.822562 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.822717 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:25:46.823554 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.824578 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.826528 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.827401 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.827573 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.827658 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.828000 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.828139 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260808 02:25:46.828676 31720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.815925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41151 (local address 127.19.231.1:37629)
0808 02:25:46.822192 (+ 6267us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:46.822212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:46.822345 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:46.822896 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:46.822905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:46.822936 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:46.823353 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.823367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.824695 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.824701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.826374 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.826387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.827654 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.827660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.827848 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.827871 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:25:46.827974 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:25:46.828123 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6143,"mutex_wait_us":25,"thread_start_us":6300,"threads_started":1}
I20260808 02:25:46.828722 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.815698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:44435 (local address 127.19.231.1:36215)
0808 02:25:46.821443 (+ 5745us) negotiation.cc:107] Waiting for socket to connect
0808 02:25:46.821464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:25:46.821612 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:25:46.822467 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:25:46.822472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:25:46.822498 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:25:46.822967 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.822986 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.824624 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.824632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.826377 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:25:46.826390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.827615 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.827622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:25:46.827784 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.827847 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 02:25:46.827946 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 02:25:46.828107 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5589,"mutex_wait_us":41,"thread_start_us":6190,"threads_started":1}
I20260808 02:25:46.829845 31507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4159a525a3024e24ab916c7d80c0eb50" candidate_uuid: "1d0ae2a95c06457589baea3b342d922d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e59eb10a8b34513bfc212912fd42edc" is_pre_election: true
I20260808 02:25:46.829845 31646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4159a525a3024e24ab916c7d80c0eb50" candidate_uuid: "1d0ae2a95c06457589baea3b342d922d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8123a7b2127431eb49028b6879515bd" is_pre_election: true
I20260808 02:25:46.830407 31646 raft_consensus.cc:2454] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0ae2a95c06457589baea3b342d922d in term 0.
I20260808 02:25:46.830413 31507 raft_consensus.cc:2454] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0ae2a95c06457589baea3b342d922d in term 0.
I20260808 02:25:46.831187 31297 leader_election.cc:302] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d0ae2a95c06457589baea3b342d922d, 5e59eb10a8b34513bfc212912fd42edc, b8123a7b2127431eb49028b6879515bd; no voters:
I20260808 02:25:46.831609 31719 raft_consensus.cc:2781] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:25:46.831765 31719 raft_consensus.cc:482] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:25:46.831877 31719 raft_consensus.cc:3037] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:46.834131 31719 raft_consensus.cc:504] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.834878 31719 leader_election.cc:288] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [CANDIDATE]: Term 1 election: Requested vote from peers 5e59eb10a8b34513bfc212912fd42edc (127.19.231.2:41151), b8123a7b2127431eb49028b6879515bd (127.19.231.3:44435)
I20260808 02:25:46.835243 31507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4159a525a3024e24ab916c7d80c0eb50" candidate_uuid: "1d0ae2a95c06457589baea3b342d922d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e59eb10a8b34513bfc212912fd42edc"
I20260808 02:25:46.835356 31646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4159a525a3024e24ab916c7d80c0eb50" candidate_uuid: "1d0ae2a95c06457589baea3b342d922d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8123a7b2127431eb49028b6879515bd"
I20260808 02:25:46.835467 31507 raft_consensus.cc:3037] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:46.835551 31646 raft_consensus.cc:3037] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:25:46.838861 31646 raft_consensus.cc:2454] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0ae2a95c06457589baea3b342d922d in term 1.
I20260808 02:25:46.838861 31507 raft_consensus.cc:2454] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0ae2a95c06457589baea3b342d922d in term 1.
I20260808 02:25:46.839296 31297 leader_election.cc:302] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0ae2a95c06457589baea3b342d922d, b8123a7b2127431eb49028b6879515bd; no voters:
I20260808 02:25:46.839651 31719 raft_consensus.cc:2781] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:25:46.839818 31719 raft_consensus.cc:686] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 1 LEADER]: Becoming Leader. State: Replica: 1d0ae2a95c06457589baea3b342d922d, State: Running, Role: LEADER
I20260808 02:25:46.840306 31719 consensus_queue.cc:237] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } }
I20260808 02:25:46.844616 31219 catalog_manager.cc:5495] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0ae2a95c06457589baea3b342d922d (127.19.231.1). New cstate: current_term: 1 leader_uuid: "1d0ae2a95c06457589baea3b342d922d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d0ae2a95c06457589baea3b342d922d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 } health_report { overall_health: UNKNOWN } } }
I20260808 02:25:46.870177 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.864737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.19.231.1:36701)
0808 02:25:46.864954 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.864961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.864980 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:25:46.865098 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.865102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.865193 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.865348 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:25:46.865730 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.866489 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.867946 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.868384 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.868519 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.868999 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:25:46.869545 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:25:46.869697 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.869786 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.869869 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 02:25:46.870278 31554 tablet.cc:2369] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:25:46.878381 31507 raft_consensus.cc:1261] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 1 FOLLOWER]: Refusing update from remote peer 1d0ae2a95c06457589baea3b342d922d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:46.878381 31646 raft_consensus.cc:1261] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 1 FOLLOWER]: Refusing update from remote peer 1d0ae2a95c06457589baea3b342d922d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:25:46.879380 31717 consensus_queue.cc:1035] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e59eb10a8b34513bfc212912fd42edc" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:46.879732 31719 consensus_queue.cc:1035] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8123a7b2127431eb49028b6879515bd" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:25:46.908411 31220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:46.908712 31220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:25:46.912833 31220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b3363f78c504190a1128ebe782ddfb4: Sending DeleteTablet for 3 replicas of tablet 4159a525a3024e24ab916c7d80c0eb50
I20260808 02:25:46.913997 31486 tablet_service.cc:1514] Processing DeleteTablet for tablet 4159a525a3024e24ab916c7d80c0eb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:46 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:46258
I20260808 02:25:46.914245 31623 tablet_service.cc:1514] Processing DeleteTablet for tablet 4159a525a3024e24ab916c7d80c0eb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:46 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:58792
I20260808 02:25:46.915127 31348 tablet_service.cc:1514] Processing DeleteTablet for tablet 4159a525a3024e24ab916c7d80c0eb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:25:46 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:53782
I20260808 02:25:46.920112 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 31281
I20260808 02:25:46.935698 31737 tablet_replica.cc:330] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: stopping tablet replica
I20260808 02:25:46.936335 31737 raft_consensus.cc:2229] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:25:46.937211 31737 raft_consensus.cc:2258] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:46.938042 31736 tablet_replica.cc:330] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: stopping tablet replica
I20260808 02:25:46.938532 31736 raft_consensus.cc:2229] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:46.939168 31736 raft_consensus.cc:2258] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:46.941376 31736 ts_tablet_manager.cc:1905] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:46.941383 31737 ts_tablet_manager.cc:1905] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:46.948782 31736 ts_tablet_manager.cc:1918] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:46.949055 31736 log.cc:1196] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/4159a525a3024e24ab916c7d80c0eb50
I20260808 02:25:46.949669 31736 ts_tablet_manager.cc:1939] T 4159a525a3024e24ab916c7d80c0eb50 P b8123a7b2127431eb49028b6879515bd: Deleting consensus metadata
I20260808 02:25:46.950904 31735 tablet_replica.cc:330] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: stopping tablet replica
I20260808 02:25:46.951372 31735 raft_consensus.cc:2229] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:25:46.951979 31735 raft_consensus.cc:2258] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:25:46.951900 31204 catalog_manager.cc:4841] TS b8123a7b2127431eb49028b6879515bd (127.19.231.3:44435): tablet 4159a525a3024e24ab916c7d80c0eb50 (table test-table [id=f97002f775df4e0ca45d4e744244bd7b]) successfully deleted
I20260808 02:25:46.954864 31737 ts_tablet_manager.cc:1918] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:46.955140 31737 log.cc:1196] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/4159a525a3024e24ab916c7d80c0eb50
I20260808 02:25:46.955809 31737 ts_tablet_manager.cc:1939] T 4159a525a3024e24ab916c7d80c0eb50 P 1d0ae2a95c06457589baea3b342d922d: Deleting consensus metadata
I20260808 02:25:46.955931 31735 ts_tablet_manager.cc:1905] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:25:46.957700 31204 catalog_manager.cc:4841] TS 1d0ae2a95c06457589baea3b342d922d (127.19.231.1:36701): tablet 4159a525a3024e24ab916c7d80c0eb50 (table test-table [id=f97002f775df4e0ca45d4e744244bd7b]) successfully deleted
Aug 08 02:25:46 dist-test-slave-pgkr krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:25:46.963548 31735 ts_tablet_manager.cc:1918] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:25:46.963730 31735 log.cc:1196] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/4159a525a3024e24ab916c7d80c0eb50
I20260808 02:25:46.964165 31735 ts_tablet_manager.cc:1939] T 4159a525a3024e24ab916c7d80c0eb50 P 5e59eb10a8b34513bfc212912fd42edc: Deleting consensus metadata
I20260808 02:25:46.965929 31206 catalog_manager.cc:4841] TS 5e59eb10a8b34513bfc212912fd42edc (127.19.231.2:41151): tablet 4159a525a3024e24ab916c7d80c0eb50 (table test-table [id=f97002f775df4e0ca45d4e744244bd7b]) successfully deleted
I20260808 02:25:46.966977 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:46.921369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.19.231.1:36701)
0808 02:25:46.921551 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:46.921557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:46.934977 (+ 13420us) server_negotiation.cc:407] Connection header received
0808 02:25:46.935215 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:46.935223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:46.935358 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:46.935486 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:46.946990 (+ 11504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.948365 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.952417 (+ 4052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:46.953199 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:46.953315 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:46.959239 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:46.959253 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:46.959270 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:46.959311 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:46.963384 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.964293 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.964301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.964304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.964482 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:46.964947 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:46.964952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:46.964955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:46.965439 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:46.965744 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:46.966207 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 02:25:46.966454 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:25:46.967849 31410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:47.073315 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 31281
I20260808 02:25:47.096041 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 31422
Aug 08 02:25:47 dist-test-slave-pgkr krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:25:47.111436 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:47.096803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.19.231.2:41151)
0808 02:25:47.096969 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:47.096973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:47.097083 (+ 110us) server_negotiation.cc:407] Connection header received
0808 02:25:47.097259 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:47.097263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:47.097329 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:47.097462 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:47.098934 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:47.099718 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:47.100926 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:47.101309 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:47.101410 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:47.105777 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:47.105792 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:47.105808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:47.105846 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:47.108814 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:47.109453 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:47.109460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:47.109463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:47.109633 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:47.109974 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:47.109977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:47.109979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:47.110359 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:47.110571 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:47.111005 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 02:25:47.111183 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:25:47.112288 31548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:47.202556 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 31422
I20260808 02:25:47.223678 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 31559
Aug 08 02:25:47 dist-test-slave-pgkr krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:25:47.239465 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:25:47.224633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.19.231.3:44435)
0808 02:25:47.224809 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 02:25:47.224814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:25:47.224836 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:25:47.224924 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:25:47.224929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:25:47.224999 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:25:47.225089 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:25:47.226457 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:47.227359 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:47.228696 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:25:47.229126 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:25:47.229232 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:25:47.233667 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:25:47.233683 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:25:47.233698 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:25:47.233737 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:25:47.236785 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:47.237471 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:47.237479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:47.237481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:47.237667 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:25:47.238072 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:25:47.238076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:25:47.238078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:25:47.238418 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:25:47.238771 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 02:25:47.239055 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 02:25:47.239227 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:25:47.240151 31686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:47.335997 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 31559
I20260808 02:25:47.356966 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 31188
I20260808 02:25:47.358393 31250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:25:47.470474 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 31188
2026-08-08T02:25:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4492 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:25:47 dist-test-slave-pgkr krb5kdc[31745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:25:47 dist-test-slave-pgkr krb5kdc[31745](info): set up 2 sockets
Aug 08 02:25:47 dist-test-slave-pgkr krb5kdc[31745](info): commencing operation
krb5kdc: starting...
W20260808 02:25:48.632288 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:25:48 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:25:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:25:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/rangeradmin_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/rangeradmin_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/rangerlookup_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/rangerlookup_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
I20260808 02:25:49.081663 20380 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOT6xt1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/postgres -l logfile start
2026-08-08 02:25:51.596 UTC [31775] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:25:51.596 UTC [31775] LOG: listening on IPv4 address "127.19.231.20", port 48205
2026-08-08 02:25:51.600 UTC [31775] LOG: listening on Unix socket "/tmp/.s.PGSQL.48205"
2026-08-08 02:25:51.624 UTC [31778] LOG: database system was shut down at 2026-08-08 02:25:50 UTC
2026-08-08 02:25:51.628 UTC [31775] LOG: database system is ready to accept connections
I20260808 02:25:52.687729 20380 mini_postgres.cc:96] Postgres bound to 48205
2026-08-08 02:25:52.715 UTC [31787] FATAL: database "slave" does not exist
127.19.231.20:48205 - accepting connections
I20260808 02:25:52.716619 20380 mini_ranger.cc:161] Starting Ranger...
I20260808 02:25:52.752398 20380 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:25:53.252818 20380 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:25:53.252977 20380 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-admin
2026-08-08 02:25:53,592 [I] DB FLAVOR :POSTGRES
2026-08-08 02:25:53,592 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:25:53,592 [I] Checking connection..
2026-08-08 02:25:53,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:25:53,856 [I] Checking connection passed.
2026-08-08 02:25:53,857 [I] --------- Verifying version history table ---------
2026-08-08 02:25:53,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:25:54,126 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:25:54,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:25:54,359 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:25:54,359 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:25:54,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:25:54.611 UTC [31851] WARNING: there is no transaction in progress
2026-08-08 02:25:54,618 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:25:54,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:25:54,882 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:25:54,882 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:25:54,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:25:55,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:25:55,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pgkr.c.gcp-upstream.internal','N') ;"
2026-08-08 02:25:55,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:25:55,898 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:25:55,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:25:56,155 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:25:56,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:25:56,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:25:56,663 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:25:56,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:25:57.606 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.622 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.638 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.653 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.672 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.692 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.754 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.761 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.775 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.785 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.795 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.808 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.819 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.825 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.832 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:57.840 UTC [31963] WARNING: there is no transaction in progress
2026-08-08 02:25:58,493 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:25:58,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:25:58,753 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:25:58,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:25:58,998 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:25:58,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:25:59,267 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:25:59,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:25:59,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:25:59,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pgkr.c.gcp-upstream.internal';"
2026-08-08 02:26:00,013 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:26:00,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:26:00,245 [I] DB_PATCHES have already been applied
I20260808 02:26:00.251363 20380 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOT6xt1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:26:00.251494 20380 mini_ranger.cc:193] Using host: 127.19.231.20
I20260808 02:26:00.275494 20380 mini_ranger.cc:227] Ranger admin URL: http://127.19.231.20:54191
Aug 08, 2026 2:26:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:26:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:26:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:26:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:26:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:26:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:26:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:26:01.486768 20380 mini_ranger.cc:160] Time spent starting Ranger: real 8.770s user 0.006s sys 0.065s
Aug 08, 2026 2:26:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:26:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 2:26:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:26:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:26:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:26:23.863982 20380 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/rangerkms_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/rangerkms_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.231.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.231.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:26:24 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:26:24.089615 20380 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:26:24.120213 20380 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:26:25.003065 20380 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:26:25.003264 20380 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms
I20260808 02:26:25.003382 20380 mini_ranger_kms.cc:227] Using postgres at 127.19.231.20:48205
2026-08-08 02:26:25,502 [I] DB FLAVOR :POSTGRES
2026-08-08 02:26:25,502 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:26:25,502 [I] Checking connection
2026-08-08 02:26:25,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:26:25,740 [I] connection success
2026-08-08 02:26:25,741 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:26:25,741 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:26:25,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:26:25,974 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:26:25,974 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:26:25,974 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:26:25,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:48205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:26:26,252 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:26:26.746979 20380 mini_ranger_kms.cc:310] Created kms service
I20260808 02:26:26.927835 20380 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:26:26.972499 20380 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:26:26 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155984, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:26:27.469048 20380 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:26:27.473132 20380 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:26:27.473232 20380 mini_ranger_kms.cc:240] Using host: 127.19.231.20
I20260808 02:26:27.501546 20380 mini_ranger_kms.cc:276] Ranger KMS PID: 32150
I20260808 02:26:27.501770 20380 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.231.20:34927
Aug 08, 2026 2:26:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:26:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:26:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:26:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:26:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:26:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:26:29 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.231.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:30.007808 20380 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.918s user 0.153s sys 0.391s
I20260808 02:26:30.008208 20380 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:26:30.008471 20380 mini_ranger_kms.cc:401] 127.19.231.20:34927/kms/v1/keys
Aug 08 02:26:30 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.231.20@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
Aug 08 02:26:30 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.231.20@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
Aug 08 02:26:33 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:26:33.107777 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40403
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:40403
--ranger_config_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:34927/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:33.799949 32202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:33.800279 32202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:33.800496 32202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:33.811133 32202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:26:33.811353 32202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:33.811553 32202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:33.811730 32202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:26:33.811849 32202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:26:33.826972 32202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.231.20:34927/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:40403
--ranger_config_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:26:33.828639 32202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:33.830220 32202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:33.850093 32207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:33.850337 32210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:33.850418 32208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:33.852025 32202 server_base.cc:1034] running on GCE node
I20260808 02:26:33.853066 32202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:33.854902 32202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:33.856194 32202 hybrid_clock.cc:648] HybridClock initialized: now 1786155993856106 us; error 49 us; skew 500 ppm
I20260808 02:26:33.856703 32202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:26:33 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:33.861393 32202 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:26:33.863700 32202 webserver.cc:460] Webserver started at http://127.19.231.62:34299/ using document root <none> and password file <none>
I20260808 02:26:33.864475 32202 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:33.864662 32202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:33.864998 32202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:26:33 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:26:33.937551 32202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "37e15f8491e4485486bdff1bfa308b72"
format_stamp: "Formatted at 2026-08-08 02:26:33 on dist-test-slave-pgkr"
server_key: "c158158414ffd1ce2651b2cf3edae4dd"
server_key_iv: "fdf1f57f6b171f2811a7f6b4d7dcfd4e"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:33.939138 32202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "37e15f8491e4485486bdff1bfa308b72"
format_stamp: "Formatted at 2026-08-08 02:26:33 on dist-test-slave-pgkr"
server_key: "c158158414ffd1ce2651b2cf3edae4dd"
server_key_iv: "fdf1f57f6b171f2811a7f6b4d7dcfd4e"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:33.949421 32202 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.008s
I20260808 02:26:33.976207 32217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:33.978055 32202 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260808 02:26:33.978343 32202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "37e15f8491e4485486bdff1bfa308b72"
format_stamp: "Formatted at 2026-08-08 02:26:33 on dist-test-slave-pgkr"
server_key: "c158158414ffd1ce2651b2cf3edae4dd"
server_key_iv: "fdf1f57f6b171f2811a7f6b4d7dcfd4e"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:33.978708 32202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:33.998983 32202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:34.005298 32202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:34.005738 32202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:34.063604 32202 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:40403
I20260808 02:26:34.063714 32270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:40403 every 8 connection(s)
I20260808 02:26:34.065562 32202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:26:34.067844 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 32202
I20260808 02:26:34.068095 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:26:34.069084 20380 external_mini_cluster.cc:1428] Setting key eb723fae3ed5fbe40c7b98e514f0cef7
I20260808 02:26:34.072328 32271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:26:34 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:26:34.091411 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72: Bootstrap starting.
I20260808 02:26:34.096719 32271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:34.099046 32271 log.cc:824] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:34.099318 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:34.071765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.19.231.62:40403)
0808 02:26:34.074119 (+ 2354us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:34.074138 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:34.074213 (+ 75us) server_negotiation.cc:407] Connection header received
0808 02:26:34.076407 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:34.076431 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:34.076873 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:34.077212 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:34.079107 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:34.081177 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:34.083155 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:34.084121 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:34.084325 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:34.089455 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:34.089465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:34.089480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:34.089522 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:34.094100 (+ 4578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:34.094926 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:34.094934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:34.094941 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:34.095125 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:34.095599 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:34.095603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:34.095605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:34.096323 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:34.096779 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:34.097260 (+ 481us) server_negotiation.cc:299] Negotiation successful
0808 02:26:34.097718 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2055,"thread_start_us":1768,"threads_started":1}
I20260808 02:26:34.104038 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72: No bootstrap required, opened a new log
I20260808 02:26:34.115460 32271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e15f8491e4485486bdff1bfa308b72" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40403 } }
I20260808 02:26:34.115999 32271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:34.116186 32271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e15f8491e4485486bdff1bfa308b72, State: Initialized, Role: FOLLOWER
I20260808 02:26:34.116894 32271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e15f8491e4485486bdff1bfa308b72" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40403 } }
I20260808 02:26:34.117221 32271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:26:34.117504 32271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:26:34.117831 32271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:34.121075 32271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e15f8491e4485486bdff1bfa308b72" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40403 } }
I20260808 02:26:34.121692 32271 leader_election.cc:302] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e15f8491e4485486bdff1bfa308b72; no voters:
I20260808 02:26:34.123013 32271 leader_election.cc:288] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:26:34.123183 32276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:34.125188 32276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [term 1 LEADER]: Becoming Leader. State: Replica: 37e15f8491e4485486bdff1bfa308b72, State: Running, Role: LEADER
I20260808 02:26:34.125941 32276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e15f8491e4485486bdff1bfa308b72" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40403 } }
I20260808 02:26:34.126406 32271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:26:34.130118 32277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37e15f8491e4485486bdff1bfa308b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e15f8491e4485486bdff1bfa308b72" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40403 } } }
I20260808 02:26:34.130960 32277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:34.131750 32278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e15f8491e4485486bdff1bfa308b72. Latest consensus state: current_term: 1 leader_uuid: "37e15f8491e4485486bdff1bfa308b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e15f8491e4485486bdff1bfa308b72" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40403 } } }
I20260808 02:26:34.132521 32278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72 [sys.catalog]: This master's current role is: LEADER
I20260808 02:26:34.134779 32282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:26:34.146283 32282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:26:34.155457 32271 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:26:34.178660 32282 catalog_manager.cc:1347] Generated new cluster ID: 9a946a2bbe014ea195547fb612989420
I20260808 02:26:34.178881 32282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:26:34.206430 32282 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:26:34.209259 32282 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:26:34.250746 32282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72: Generated new TSK 0
I20260808 02:26:34.252207 32282 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:26:35.203140 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:34927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40403
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:26:35 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786155995, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:26:35.965662 32319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:35.966212 32319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:35.966475 32319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:35.985441 32319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:35.985755 32319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:35.986119 32319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:26:36.010587 32319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.231.20:34927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:26:36.012974 32319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:36.015406 32319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:36.085767 32329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:36.086516 32330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:36.110988 32319 server_base.cc:1034] running on GCE node
W20260808 02:26:36.112443 32332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:36.113584 32319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:36.119195 32319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:36.120656 32319 hybrid_clock.cc:648] HybridClock initialized: now 1786155996120429 us; error 128 us; skew 500 ppm
I20260808 02:26:36.121245 32319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:26:36 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155996, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:36.127568 32319 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:26:36.134871 32319 webserver.cc:460] Webserver started at http://127.19.231.1:33455/ using document root <none> and password file <none>
I20260808 02:26:36.135816 32319 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:36.136046 32319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:36.136543 32319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:26:36 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155996, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:26:36.163851 32319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "934d5c6991524c77bbf7ff1569007672"
format_stamp: "Formatted at 2026-08-08 02:26:36 on dist-test-slave-pgkr"
server_key: "dfc46c1f76d503bc920231b1fdc3c20a"
server_key_iv: "ab0c0858d8e659d77e62ccf1b423b43e"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:36.165269 32319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "934d5c6991524c77bbf7ff1569007672"
format_stamp: "Formatted at 2026-08-08 02:26:36 on dist-test-slave-pgkr"
server_key: "dfc46c1f76d503bc920231b1fdc3c20a"
server_key_iv: "ab0c0858d8e659d77e62ccf1b423b43e"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:36.184222 32319 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.014s sys 0.002s
I20260808 02:26:36.228746 32339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:36.232573 32319 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.008s sys 0.004s
I20260808 02:26:36.232796 32319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "934d5c6991524c77bbf7ff1569007672"
format_stamp: "Formatted at 2026-08-08 02:26:36 on dist-test-slave-pgkr"
server_key: "dfc46c1f76d503bc920231b1fdc3c20a"
server_key_iv: "ab0c0858d8e659d77e62ccf1b423b43e"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:36.233047 32319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 02:26:36 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155995, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:26:36.292807 32319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:36.300909 32319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:36.301584 32319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:36.350934 32319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:36.356827 32319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:36.357030 32319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:36.357306 32319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:36.357473 32319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:26:36 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155995, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:26:36.727540 32455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:36625 every 8 connection(s)
I20260808 02:26:36.738881 32319 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:36625
I20260808 02:26:36.741392 32319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:26:36.744876 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 32319
I20260808 02:26:36.745085 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:26:36.746183 20380 external_mini_cluster.cc:1428] Setting key f5ee46355cff2996b8281b9bd7e9e820
Aug 08 02:26:36 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155996, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:26:36.798254 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:36.747364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:36717 (local address 127.19.231.62:40403)
0808 02:26:36.760496 (+ 13132us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:36.760505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:36.763331 (+ 2826us) server_negotiation.cc:407] Connection header received
0808 02:26:36.768454 (+ 5123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:36.768465 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:36.768594 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:36.768695 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:36.771954 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:36.773237 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:36.776375 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:36.777114 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:36.777250 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:36.788004 (+ 10754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:36.788017 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:36.788020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:36.788059 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:36.793398 (+ 5339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:36.794902 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:36.794909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:36.794913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:36.795072 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:36.795737 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:36.795746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:36.795748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:36.796157 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:36.796333 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:36.797406 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0808 02:26:36.797637 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12916,"thread_start_us":13983,"threads_started":1}
I20260808 02:26:36.799475 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:36.746424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40403 (local address 127.19.231.1:36717)
0808 02:26:36.763124 (+ 16700us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:36.763202 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:36.768049 (+ 4847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:36.769477 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:36.769492 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:36.770520 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:36.771725 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:36.771758 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:36.773492 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:36.773499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:36.776183 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:36.776197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:36.777946 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:36.777957 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:36.778337 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:36.779475 (+ 1138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:36.779507 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:36.787647 (+ 8140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:36.793809 (+ 6162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:36.793819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:36.793841 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:36.794771 (+ 930us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:36.795202 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:36.795207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:36.795210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:36.795606 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:36.796304 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:36.796313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:36.796868 (+ 555us) client_negotiation.cc:769] Sending connection context
0808 02:26:36.797241 (+ 373us) client_negotiation.cc:240] Negotiation successful
0808 02:26:36.797888 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16254,"thread_start_us":14597,"threads_started":1}
I20260808 02:26:36.801505 32456 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40403
I20260808 02:26:36.801890 32456 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:36.803110 32456 heartbeater.cc:511] Master 127.19.231.62:40403 requested a full tablet report, sending...
I20260808 02:26:36.805866 32234 ts_manager.cc:194] Registered new tserver with Master: 934d5c6991524c77bbf7ff1569007672 (127.19.231.1:36625)
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
I20260808 02:26:36.808434 32234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:36717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
W20260808 02:26:36.859196 20380 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.19.231.2: real 0.109s user 0.000s sys 0.027s
I20260808 02:26:36.860270 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:34927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40403
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:37.621218 32464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:37.621565 32464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:37.621754 32464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:37.632032 32464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:37.632318 32464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:37.632624 32464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:26:37.647920 32464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.231.20:34927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:26:37.649701 32464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:37.651413 32464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:37.677887 32477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:37.679298 32478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:37.680112 32480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:37.684106 32464 server_base.cc:1034] running on GCE node
I20260808 02:26:37.685060 32464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:37.686430 32464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:37.687796 32464 hybrid_clock.cc:648] HybridClock initialized: now 1786155997687560 us; error 103 us; skew 500 ppm
I20260808 02:26:37.688198 32464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:26:37 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:37.693862 32464 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:26:37.696785 32464 webserver.cc:460] Webserver started at http://127.19.231.2:35175/ using document root <none> and password file <none>
I20260808 02:26:37.697819 32464 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:37.698051 32464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:37.698508 32464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:26:37 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:26:37.728180 32464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "02c8fa1e53d6446db85bdf8fc5faea75"
format_stamp: "Formatted at 2026-08-08 02:26:37 on dist-test-slave-pgkr"
server_key: "23c9e515069789a0ba03772661e0304d"
server_key_iv: "fc85b6dc26c456dc3798680588618cc7"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:37.729758 32464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "02c8fa1e53d6446db85bdf8fc5faea75"
format_stamp: "Formatted at 2026-08-08 02:26:37 on dist-test-slave-pgkr"
server_key: "23c9e515069789a0ba03772661e0304d"
server_key_iv: "fc85b6dc26c456dc3798680588618cc7"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:37.738790 32464 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20260808 02:26:37.762560 32488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:37.764918 32464 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260808 02:26:37.765205 32464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "02c8fa1e53d6446db85bdf8fc5faea75"
format_stamp: "Formatted at 2026-08-08 02:26:37 on dist-test-slave-pgkr"
server_key: "23c9e515069789a0ba03772661e0304d"
server_key_iv: "fc85b6dc26c456dc3798680588618cc7"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:37.765589 32464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:37.807657 32464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:37.812866 32456 heartbeater.cc:503] Master 127.19.231.62:40403 was elected leader, sending a full tablet report...
I20260808 02:26:37.815716 32464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:37.816393 32464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:37.818696 32464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:37.821941 32464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:37.822124 32464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:37.822404 32464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:37.822656 32464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:37.965242 32464 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:40747
I20260808 02:26:37.965317 32602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:40747 every 8 connection(s)
I20260808 02:26:37.966866 32464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:26:37.973527 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 32464
I20260808 02:26:37.973750 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:26:37.974267 20380 external_mini_cluster.cc:1428] Setting key 09e3cf3f2cbda38a90295d0c4bca1a67
Aug 08 02:26:37 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:26:38.000576 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:37.970506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:40131 (local address 127.19.231.62:40403)
0808 02:26:37.971446 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:37.971454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:37.972809 (+ 1355us) server_negotiation.cc:407] Connection header received
0808 02:26:37.976288 (+ 3479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:37.976295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:37.976389 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:37.976514 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:37.978542 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.979792 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.982376 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.983124 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.983262 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:37.992561 (+ 9299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:37.992568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:37.992573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:37.992615 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:37.996486 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:37.997593 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:37.997599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:37.997602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:37.997758 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:37.998313 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:37.998318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:37.998322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:37.998719 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:37.998864 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:37.999956 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0808 02:26:38.000213 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":553,"threads_started":1}
I20260808 02:26:38.002141 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:37.971342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40403 (local address 127.19.231.2:40131)
0808 02:26:37.972606 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:37.972669 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:37.975664 (+ 2995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:37.976804 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:37.976817 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:37.977437 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:37.978338 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:37.978365 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.980000 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.980006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:37.982213 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:37.982229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:37.983428 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:37.983436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:37.983731 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:37.989073 (+ 5342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:37.989120 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:37.992276 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:37.996659 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:37.996672 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:37.996693 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:37.997452 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:37.997897 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:37.997904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:37.997910 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:37.998188 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:37.998895 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:37.998907 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:37.999421 (+ 514us) client_negotiation.cc:769] Sending connection context
0808 02:26:37.999827 (+ 406us) client_negotiation.cc:240] Negotiation successful
0808 02:26:38.000487 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":631,"threads_started":1}
I20260808 02:26:38.004158 32603 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40403
I20260808 02:26:38.004505 32603 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:38.005443 32603 heartbeater.cc:511] Master 127.19.231.62:40403 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
I20260808 02:26:38.007289 32234 ts_manager.cc:194] Registered new tserver with Master: 02c8fa1e53d6446db85bdf8fc5faea75 (127.19.231.2:40747)
I20260808 02:26:38.008394 32234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:40131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:26:38.045885 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:34927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40403
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:26:38.713694 32609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:26:38.714095 32609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:26:38.714386 32609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:26:38.724987 32609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:26:38.725225 32609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:26:38.725499 32609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:26:38.739863 32609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.231.20:34927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:26:38.741526 32609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:26:38.743144 32609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:26:38.765470 32614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:38.765482 32617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:26:38.765533 32615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:26:38.767827 32609 server_base.cc:1034] running on GCE node
I20260808 02:26:38.768533 32609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:26:38.769652 32609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:26:38.770859 32609 hybrid_clock.cc:648] HybridClock initialized: now 1786155998770821 us; error 36 us; skew 500 ppm
I20260808 02:26:38.771220 32609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:26:38 dist-test-slave-pgkr krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:26:38.775089 32609 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:26:38.777056 32609 webserver.cc:460] Webserver started at http://127.19.231.3:36993/ using document root <none> and password file <none>
I20260808 02:26:38.777788 32609 fs_manager.cc:359] Metadata directory not provided
I20260808 02:26:38.777906 32609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:26:38.778223 32609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:26:38 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:26:38.796149 32609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "ec843be6fab445899769b2599441f3d0"
format_stamp: "Formatted at 2026-08-08 02:26:38 on dist-test-slave-pgkr"
server_key: "500ba13d6fdc399af50a0bdc715a0b5c"
server_key_iv: "2da31549d91a68f0ea532f8e5e17bf00"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:38.797179 32609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "ec843be6fab445899769b2599441f3d0"
format_stamp: "Formatted at 2026-08-08 02:26:38 on dist-test-slave-pgkr"
server_key: "500ba13d6fdc399af50a0bdc715a0b5c"
server_key_iv: "2da31549d91a68f0ea532f8e5e17bf00"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:38.804948 32609 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260808 02:26:38.826845 32624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:38.829252 32609 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260808 02:26:38.829555 32609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "ec843be6fab445899769b2599441f3d0"
format_stamp: "Formatted at 2026-08-08 02:26:38 on dist-test-slave-pgkr"
server_key: "500ba13d6fdc399af50a0bdc715a0b5c"
server_key_iv: "2da31549d91a68f0ea532f8e5e17bf00"
server_key_version: "kuduclusterkey@0"
I20260808 02:26:38.829974 32609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:26:38.856184 32609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:26:38.864704 32609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:26:38.865429 32609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:26:38.867712 32609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:26:38.871040 32609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:26:38.871238 32609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:38.871505 32609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:26:38.871722 32609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:26:38.940060 32609 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:43309
I20260808 02:26:38.940142 32742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:43309 every 8 connection(s)
I20260808 02:26:38.941722 32609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:26:38.947489 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 32609
I20260808 02:26:38.947722 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:26:38.948560 20380 external_mini_cluster.cc:1428] Setting key 7a218b1745f613b0df2021f65b702176
Aug 08 02:26:38 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:26:38.973096 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:38.945242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:42101 (local address 127.19.231.62:40403)
0808 02:26:38.945937 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:38.945943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:38.947892 (+ 1949us) server_negotiation.cc:407] Connection header received
0808 02:26:38.952110 (+ 4218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:38.952123 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:38.952219 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:38.952388 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:38.954676 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.955886 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.958259 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.959120 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.959270 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:38.964364 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:38.964373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:38.964378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:38.964439 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:38.968972 (+ 4533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:38.970159 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:38.970164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:38.970166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:38.970379 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:38.971089 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:38.971093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:38.971096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:38.971460 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:38.971570 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:38.972503 (+ 933us) server_negotiation.cc:299] Negotiation successful
0808 02:26:38.972729 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":422,"threads_started":1}
I20260808 02:26:38.974272 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:38.946043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40403 (local address 127.19.231.3:42101)
0808 02:26:38.947714 (+ 1671us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:38.947777 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:38.951825 (+ 4048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:38.952698 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:38.952707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:38.953519 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:26:38.954477 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:38.954498 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.956104 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.956110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:38.958064 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:38.958078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:38.959509 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:38.959521 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:38.959807 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:38.960571 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:26:38.960596 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:26:38.964027 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:26:38.969120 (+ 5093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:38.969130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:38.969159 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:38.970010 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:38.970520 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:26:38.970528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:26:38.970532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:26:38.970975 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:26:38.971634 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:26:38.971640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:26:38.971971 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 02:26:38.972407 (+ 436us) client_negotiation.cc:240] Negotiation successful
0808 02:26:38.972880 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"thread_start_us":828,"threads_started":1}
I20260808 02:26:38.975749 32743 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40403
I20260808 02:26:38.976099 32743 heartbeater.cc:464] Registering TS with master...
I20260808 02:26:38.977581 32743 heartbeater.cc:511] Master 127.19.231.62:40403 requested a full tablet report, sending...
I20260808 02:26:38.979588 32234 ts_manager.cc:194] Registered new tserver with Master: ec843be6fab445899769b2599441f3d0 (127.19.231.3:43309)
I20260808 02:26:38.980679 32234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:42101
I20260808 02:26:38.991024 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:26:39.012056 32603 heartbeater.cc:503] Master 127.19.231.62:40403 was elected leader, sending a full tablet report...
I20260808 02:26:39.017746 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:38.998262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.19.231.62:40403)
0808 02:26:38.998685 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:38.998695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:38.999412 (+ 717us) server_negotiation.cc:407] Connection header received
0808 02:26:38.999681 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:38.999687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:38.999879 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.000004 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:39.002363 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.003587 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.005770 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.006594 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.006799 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.010118 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:39.010124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:39.010127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:39.010162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:39.014728 (+ 4566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.015565 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.015568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.015571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.015752 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.016169 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.016172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.016174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.016453 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:39.016713 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.016997 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.017341 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260808 02:26:39.022567 32234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:26:39.035086 32234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:26:39.085846 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.052073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.19.231.1:36625)
0808 02:26:39.065457 (+ 13384us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.065473 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.068817 (+ 3344us) server_negotiation.cc:407] Connection header received
0808 02:26:39.070463 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.070472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.070795 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.071101 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:39.073006 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.075725 (+ 2719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.080321 (+ 4596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.082906 (+ 2585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.083074 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.083479 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.084834 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.085078 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13042,"thread_start_us":14435,"threads_started":1}
I20260808 02:26:39.086954 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.052777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.19.231.3:43309)
0808 02:26:39.066567 (+ 13790us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.066578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.069049 (+ 2471us) server_negotiation.cc:407] Connection header received
0808 02:26:39.069277 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.069292 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.069775 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.070758 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:39.071929 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.074569 (+ 2640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.080569 (+ 6000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.082950 (+ 2381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.083084 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.083451 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.085951 (+ 2500us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.086200 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13503,"thread_start_us":13226,"threads_started":1}
I20260808 02:26:39.087548 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.052085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:43309 (local address 127.0.0.1:49862)
0808 02:26:39.068583 (+ 16498us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:39.068664 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:39.068959 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:39.070177 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:39.070191 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:39.070244 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:39.071453 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.071475 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.074777 (+ 3302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.074784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.080315 (+ 5531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.080335 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.083873 (+ 3538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.083879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.084332 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.084485 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 02:26:39.084671 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 02:26:39.084906 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16208,"mutex_wait_us":138,"thread_start_us":15548,"threads_started":1}
I20260808 02:26:39.087606 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.051770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:36625 (local address 127.0.0.1:54370)
0808 02:26:39.068583 (+ 16813us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:39.068664 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:39.070178 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:39.071582 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:39.071586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:39.071608 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:39.072633 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.072652 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.075990 (+ 3338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.075996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.080106 (+ 4110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.080121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.083809 (+ 3688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.083819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.084343 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.084485 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 02:26:39.084671 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 02:26:39.085016 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16326,"mutex_wait_us":177,"thread_start_us":15970,"threads_started":1}
I20260808 02:26:39.089073 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.069225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.19.231.2:40747)
0808 02:26:39.070379 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.070389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.076195 (+ 5806us) server_negotiation.cc:407] Connection header received
0808 02:26:39.076324 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.076330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.076565 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.076891 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:39.079040 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.081136 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.084486 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.086695 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.086916 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.087247 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.088468 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.088664 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":712,"threads_started":1}
I20260808 02:26:39.089252 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.068963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:40747 (local address 127.0.0.1:39086)
0808 02:26:39.075887 (+ 6924us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:39.075911 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:39.076088 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:39.076907 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:39.076915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:39.076948 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:39.078735 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.078758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.081365 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.081371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.084307 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.084320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.087777 (+ 3457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.087788 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.088173 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.088219 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:26:39.088369 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 02:26:39.088507 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6743,"mutex_wait_us":24,"thread_start_us":6397,"threads_started":1}
I20260808 02:26:39.093251 32674 tablet_service.cc:1467] Processing CreateTablet for tablet 32c40a707b5342ffa1d4f5d57e90a7f0 (DEFAULT_TABLE table=test-table [id=54afb7c4bc854b158f5ed422c6292d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:39.094389 32536 tablet_service.cc:1467] Processing CreateTablet for tablet 32c40a707b5342ffa1d4f5d57e90a7f0 (DEFAULT_TABLE table=test-table [id=54afb7c4bc854b158f5ed422c6292d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:39.094359 32389 tablet_service.cc:1467] Processing CreateTablet for tablet 32c40a707b5342ffa1d4f5d57e90a7f0 (DEFAULT_TABLE table=test-table [id=54afb7c4bc854b158f5ed422c6292d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:26:39.095609 32674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c40a707b5342ffa1d4f5d57e90a7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:39.096355 32536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c40a707b5342ffa1d4f5d57e90a7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:39.096380 32389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c40a707b5342ffa1d4f5d57e90a7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:26:39.125164 32762 tablet_bootstrap.cc:492] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Bootstrap starting.
I20260808 02:26:39.129098 32763 tablet_bootstrap.cc:492] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Bootstrap starting.
I20260808 02:26:39.135619 32762 tablet_bootstrap.cc:654] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:39.137844 32762 log.cc:824] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:39.139618 32764 tablet_bootstrap.cc:492] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Bootstrap starting.
I20260808 02:26:39.142894 32762 tablet_bootstrap.cc:492] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: No bootstrap required, opened a new log
I20260808 02:26:39.143477 32762 ts_tablet_manager.cc:1397] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20260808 02:26:39.146023 32764 tablet_bootstrap.cc:654] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:39.148375 32763 tablet_bootstrap.cc:654] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Neither blocks nor log segments found. Creating new log.
I20260808 02:26:39.148424 32764 log.cc:824] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:39.150799 32763 log.cc:824] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Log is configured to *not* fsync() on all Append() calls
I20260808 02:26:39.152612 32764 tablet_bootstrap.cc:492] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: No bootstrap required, opened a new log
I20260808 02:26:39.153010 32764 ts_tablet_manager.cc:1397] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260808 02:26:39.155540 32763 tablet_bootstrap.cc:492] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: No bootstrap required, opened a new log
I20260808 02:26:39.155911 32763 ts_tablet_manager.cc:1397] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20260808 02:26:39.161156 32762 raft_consensus.cc:348] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.161872 32762 raft_consensus.cc:374] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:39.162031 32762 raft_consensus.cc:729] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec843be6fab445899769b2599441f3d0, State: Initialized, Role: FOLLOWER
I20260808 02:26:39.163106 32762 consensus_queue.cc:260] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.163828 32764 raft_consensus.cc:348] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.164604 32764 raft_consensus.cc:374] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:39.164777 32764 raft_consensus.cc:729] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934d5c6991524c77bbf7ff1569007672, State: Initialized, Role: FOLLOWER
I20260808 02:26:39.165302 32743 heartbeater.cc:503] Master 127.19.231.62:40403 was elected leader, sending a full tablet report...
I20260808 02:26:39.165992 32762 ts_tablet_manager.cc:1428] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20260808 02:26:39.165781 32764 consensus_queue.cc:260] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.168519 32764 ts_tablet_manager.cc:1428] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260808 02:26:39.172420 32763 raft_consensus.cc:348] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.173156 32763 raft_consensus.cc:374] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:26:39.173336 32763 raft_consensus.cc:729] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c8fa1e53d6446db85bdf8fc5faea75, State: Initialized, Role: FOLLOWER
I20260808 02:26:39.174170 32763 consensus_queue.cc:260] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.176718 32763 ts_tablet_manager.cc:1428] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Time spent starting tablet: real 0.021s user 0.017s sys 0.004s
W20260808 02:26:39.194767 32744 tablet.cc:2369] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:26:39.220438 32604 tablet.cc:2369] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:39.220803 302 raft_consensus.cc:482] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:26:39.221388 302 raft_consensus.cc:504] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.224002 302 leader_election.cc:288] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 934d5c6991524c77bbf7ff1569007672 (127.19.231.1:36625), ec843be6fab445899769b2599441f3d0 (127.19.231.3:43309)
I20260808 02:26:39.235446 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.224394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:44727 (local address 127.19.231.1:36625)
0808 02:26:39.224680 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.224687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.226899 (+ 2212us) server_negotiation.cc:407] Connection header received
0808 02:26:39.227149 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.227155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.227257 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.227400 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:39.228451 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.229693 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.232722 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.234018 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.234291 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.234392 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.234860 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.235028 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 02:26:39.235844 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.224659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:35919 (local address 127.19.231.3:43309)
0808 02:26:39.224895 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.224900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.227750 (+ 2850us) server_negotiation.cc:407] Connection header received
0808 02:26:39.228009 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.228015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.228137 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.228235 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:26:39.229134 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.230079 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.232985 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.234178 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.234384 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.234478 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.235026 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.235163 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:26:39.236021 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.224764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:36625 (local address 127.19.231.2:44727)
0808 02:26:39.226682 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:39.226711 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:39.226960 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:39.227433 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:39.227437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:39.227463 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:39.228189 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.228201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.229838 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.229846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.232533 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.232546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.234387 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.234400 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.234676 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.234707 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:26:39.234848 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 02:26:39.235122 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1624,"mutex_wait_us":61,"spinlock_wait_cycles":81792,"thread_start_us":1132,"threads_started":1}
I20260808 02:26:39.236482 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.225692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:43309 (local address 127.19.231.2:35919)
0808 02:26:39.227573 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0808 02:26:39.227631 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 02:26:39.227867 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:26:39.228251 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:26:39.228255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:26:39.228278 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:26:39.228949 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.228959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.230262 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.230270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.232794 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:26:39.232807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.234562 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.234570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:26:39.234829 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.234851 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 02:26:39.234985 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 02:26:39.235187 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"mutex_wait_us":35,"spinlock_wait_cycles":896,"thread_start_us":1280,"threads_started":1}
I20260808 02:26:39.237512 32409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c40a707b5342ffa1d4f5d57e90a7f0" candidate_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934d5c6991524c77bbf7ff1569007672" is_pre_election: true
I20260808 02:26:39.237783 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c40a707b5342ffa1d4f5d57e90a7f0" candidate_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec843be6fab445899769b2599441f3d0" is_pre_election: true
I20260808 02:26:39.238442 32409 raft_consensus.cc:2454] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c8fa1e53d6446db85bdf8fc5faea75 in term 0.
I20260808 02:26:39.238730 32696 raft_consensus.cc:2454] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c8fa1e53d6446db85bdf8fc5faea75 in term 0.
I20260808 02:26:39.239657 32492 leader_election.cc:302] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c8fa1e53d6446db85bdf8fc5faea75, 934d5c6991524c77bbf7ff1569007672; no voters:
I20260808 02:26:39.240216 302 raft_consensus.cc:2781] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:26:39.240475 302 raft_consensus.cc:482] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:26:39.240672 302 raft_consensus.cc:3037] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:39.244388 302 raft_consensus.cc:504] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
I20260808 02:26:39.245666 302 leader_election.cc:288] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [CANDIDATE]: Term 1 election: Requested vote from peers 934d5c6991524c77bbf7ff1569007672 (127.19.231.1:36625), ec843be6fab445899769b2599441f3d0 (127.19.231.3:43309)
I20260808 02:26:39.246224 32409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c40a707b5342ffa1d4f5d57e90a7f0" candidate_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934d5c6991524c77bbf7ff1569007672"
I20260808 02:26:39.246568 32409 raft_consensus.cc:3037] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:39.250031 32409 raft_consensus.cc:2454] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c8fa1e53d6446db85bdf8fc5faea75 in term 1.
I20260808 02:26:39.250672 32492 leader_election.cc:302] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c8fa1e53d6446db85bdf8fc5faea75, 934d5c6991524c77bbf7ff1569007672; no voters:
I20260808 02:26:39.251168 302 raft_consensus.cc:2781] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:26:39.251242 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c40a707b5342ffa1d4f5d57e90a7f0" candidate_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec843be6fab445899769b2599441f3d0"
I20260808 02:26:39.251649 32696 raft_consensus.cc:3037] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:26:39.253587 302 raft_consensus.cc:686] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 1 LEADER]: Becoming Leader. State: Replica: 02c8fa1e53d6446db85bdf8fc5faea75, State: Running, Role: LEADER
I20260808 02:26:39.254421 302 consensus_queue.cc:237] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } }
W20260808 02:26:39.256726 32457 tablet.cc:2369] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:26:39.257198 32696 raft_consensus.cc:2454] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c8fa1e53d6446db85bdf8fc5faea75 in term 1.
I20260808 02:26:39.263550 32233 catalog_manager.cc:5495] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c8fa1e53d6446db85bdf8fc5faea75 (127.19.231.2). New cstate: current_term: 1 leader_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02c8fa1e53d6446db85bdf8fc5faea75" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260808 02:26:39.306461 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.299572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.19.231.2:40747)
0808 02:26:39.299809 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.299814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.299901 (+ 87us) server_negotiation.cc:407] Connection header received
0808 02:26:39.300098 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.300101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.300220 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.300350 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:26:39.300907 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.301878 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.303981 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.304543 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.304639 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.305336 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:26:39.305895 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:26:39.306022 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.306143 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.306227 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 02:26:39.316877 32696 raft_consensus.cc:1261] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 1 FOLLOWER]: Refusing update from remote peer 02c8fa1e53d6446db85bdf8fc5faea75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:39.317061 32409 raft_consensus.cc:1261] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 1 FOLLOWER]: Refusing update from remote peer 02c8fa1e53d6446db85bdf8fc5faea75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:26:39.318217 305 consensus_queue.cc:1035] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "934d5c6991524c77bbf7ff1569007672" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 36625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:39.318821 302 consensus_queue.cc:1035] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec843be6fab445899769b2599441f3d0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:26:39.354089 32234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:39.354553 32234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:26:39.359691 32234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37e15f8491e4485486bdff1bfa308b72: Sending DeleteTablet for 3 replicas of tablet 32c40a707b5342ffa1d4f5d57e90a7f0
I20260808 02:26:39.361202 32389 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c40a707b5342ffa1d4f5d57e90a7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:39 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:54370
I20260808 02:26:39.361337 32536 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c40a707b5342ffa1d4f5d57e90a7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:39 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:39086
I20260808 02:26:39.361308 32674 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c40a707b5342ffa1d4f5d57e90a7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:26:39 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:49862
I20260808 02:26:39.365091 317 tablet_replica.cc:330] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: stopping tablet replica
I20260808 02:26:39.365760 317 raft_consensus.cc:2229] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:39.366604 317 raft_consensus.cc:2258] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:39.367210 319 tablet_replica.cc:330] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: stopping tablet replica
I20260808 02:26:39.367651 319 raft_consensus.cc:2229] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:26:39.368115 318 tablet_replica.cc:330] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: stopping tablet replica
I20260808 02:26:39.368605 318 raft_consensus.cc:2229] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:26:39.368607 319 raft_consensus.cc:2258] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:39.369351 318 raft_consensus.cc:2258] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:26:39.370187 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 32319
I20260808 02:26:39.370183 317 ts_tablet_manager.cc:1905] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:39.374751 319 ts_tablet_manager.cc:1905] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:39.377929 317 ts_tablet_manager.cc:1918] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:39.378139 317 log.cc:1196] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/32c40a707b5342ffa1d4f5d57e90a7f0
I20260808 02:26:39.378775 317 ts_tablet_manager.cc:1939] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 934d5c6991524c77bbf7ff1569007672: Deleting consensus metadata
I20260808 02:26:39.381085 32221 catalog_manager.cc:4841] TS 934d5c6991524c77bbf7ff1569007672 (127.19.231.1:36625): tablet 32c40a707b5342ffa1d4f5d57e90a7f0 (table test-table [id=54afb7c4bc854b158f5ed422c6292d26]) successfully deleted
I20260808 02:26:39.382786 319 ts_tablet_manager.cc:1918] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:39.382962 319 log.cc:1196] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/32c40a707b5342ffa1d4f5d57e90a7f0
I20260808 02:26:39.383363 318 ts_tablet_manager.cc:1905] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:26:39.383518 319 ts_tablet_manager.cc:1939] T 32c40a707b5342ffa1d4f5d57e90a7f0 P 02c8fa1e53d6446db85bdf8fc5faea75: Deleting consensus metadata
Aug 08 02:26:39 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:26:39.385712 32221 catalog_manager.cc:4841] TS 02c8fa1e53d6446db85bdf8fc5faea75 (127.19.231.2:40747): tablet 32c40a707b5342ffa1d4f5d57e90a7f0 (table test-table [id=54afb7c4bc854b158f5ed422c6292d26]) successfully deleted
I20260808 02:26:39.392066 318 ts_tablet_manager.cc:1918] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:26:39.392279 318 log.cc:1196] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/32c40a707b5342ffa1d4f5d57e90a7f0
I20260808 02:26:39.392905 318 ts_tablet_manager.cc:1939] T 32c40a707b5342ffa1d4f5d57e90a7f0 P ec843be6fab445899769b2599441f3d0: Deleting consensus metadata
I20260808 02:26:39.395098 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.371141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.19.231.1:36625)
0808 02:26:39.371337 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.371344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.371781 (+ 437us) server_negotiation.cc:407] Connection header received
0808 02:26:39.372071 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.372078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.372168 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.372247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:39.374165 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.375392 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.377471 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.378325 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.378443 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.386256 (+ 7813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:39.386269 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:39.386284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:39.386343 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:39.390594 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.391399 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.391408 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.391412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.391677 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.392125 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.392131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.392135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.392801 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:39.393347 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.394011 (+ 664us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.394268 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:26:39.395372 32219 catalog_manager.cc:4841] TS ec843be6fab445899769b2599441f3d0 (127.19.231.3:43309): tablet 32c40a707b5342ffa1d4f5d57e90a7f0 (table test-table [id=54afb7c4bc854b158f5ed422c6292d26]) successfully deleted
I20260808 02:26:39.396209 32451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:26:39.492501 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 32319
I20260808 02:26:39.515586 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 32464
Aug 08 02:26:39 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:26:39.538928 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.516688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.19.231.2:40747)
0808 02:26:39.516909 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.516917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.516941 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:26:39.517162 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.517170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.517328 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.517452 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:39.519406 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.521286 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.522696 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.523732 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.523868 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.528565 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:39.528593 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:39.528621 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:39.528713 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:39.534969 (+ 6256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.535878 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.535895 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.535902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.536221 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.536626 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.536632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.536636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.537345 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:39.537815 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.538037 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.538322 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 02:26:39.539875 32598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:26:39.653394 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 32464
I20260808 02:26:39.677985 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 32609
Aug 08 02:26:39 dist-test-slave-pgkr krb5kdc[31745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786155993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:26:39.694321 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:26:39.678837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.19.231.3:43309)
0808 02:26:39.679022 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:26:39.679031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:26:39.679091 (+ 60us) server_negotiation.cc:407] Connection header received
0808 02:26:39.679338 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:26:39.679343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:26:39.679411 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:26:39.679520 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:26:39.680877 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.681623 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.683035 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:26:39.683443 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:26:39.683579 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:26:39.688277 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:26:39.688293 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:26:39.688309 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:26:39.688347 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:26:39.691349 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.692212 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.692227 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.692230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.692389 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:26:39.692786 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:26:39.692791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:26:39.692793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:26:39.693207 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:26:39.693443 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 02:26:39.693738 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 02:26:39.693923 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:26:39.695096 32738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:26:39.788743 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 32609
I20260808 02:26:39.810134 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 32202
I20260808 02:26:39.811502 32265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:26:39.930016 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 32202
I20260808 02:26:39.962364 20380 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:26:40.307710 20380 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:26:40.307994 20380 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:26:40.652860 20380 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:26:40.653 UTC [31775] LOG: received smart shutdown request
2026-08-08 02:26:40.657 UTC [31775] LOG: background worker "logical replication launcher" (PID 31785) exited with exit code 1
2026-08-08 02:26:40.658 UTC [31780] LOG: shutting down
2026-08-08 02:26:40.831 UTC [31775] LOG: database system is shut down
2026-08-08T02:26:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53497 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:26:41 dist-test-slave-pgkr krb5kdc[334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:26:41 dist-test-slave-pgkr krb5kdc[334](info): set up 2 sockets
Aug 08 02:26:41 dist-test-slave-pgkr krb5kdc[334](info): commencing operation
krb5kdc: starting...
W20260808 02:26:42.142884 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:26:42 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:26:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/rangeradmin_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/rangeradmin_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/rangerlookup_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/rangerlookup_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
I20260808 02:26:42.557504 20380 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOT6xt1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/postgres -l logfile start
2026-08-08 02:26:45.115 UTC [365] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 02:26:45.115 UTC [365] LOG: listening on IPv4 address "127.19.231.20", port 55461
2026-08-08 02:26:45.118 UTC [365] LOG: listening on Unix socket "/tmp/.s.PGSQL.55461"
2026-08-08 02:26:45.142 UTC [367] LOG: database system was shut down at 2026-08-08 02:26:43 UTC
2026-08-08 02:26:45.146 UTC [365] LOG: database system is ready to accept connections
I20260808 02:26:46.162352 20380 mini_postgres.cc:96] Postgres bound to 55461
2026-08-08 02:26:46.186 UTC [376] FATAL: database "slave" does not exist
127.19.231.20:55461 - accepting connections
I20260808 02:26:46.187206 20380 mini_ranger.cc:161] Starting Ranger...
I20260808 02:26:46.219730 20380 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 02:26:46.720332 20380 mini_ranger.cc:87] Created ranger Postgres database
I20260808 02:26:46.720463 20380 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-admin
2026-08-08 02:26:47,023 [I] DB FLAVOR :POSTGRES
2026-08-08 02:26:47,024 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:26:47,024 [I] Checking connection..
2026-08-08 02:26:47,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 02:26:47,276 [I] Checking connection passed.
2026-08-08 02:26:47,276 [I] --------- Verifying version history table ---------
2026-08-08 02:26:47,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:26:47,538 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:26:47,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:26:47,792 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 02:26:47,793 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 02:26:47,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 02:26:48.061 UTC [444] WARNING: there is no transaction in progress
2026-08-08 02:26:48,070 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 02:26:48,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 02:26:48,344 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 02:26:48,344 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:26:48,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 02:26:48,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 02:26:48,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pgkr.c.gcp-upstream.internal','N') ;"
2026-08-08 02:26:49,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:26:49,347 [I] Table x_portal_user does not exist in database ranger
2026-08-08 02:26:49,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:26:49,617 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 02:26:49,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:26:49,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:26:50,138 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 02:26:50,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 02:26:51.105 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.122 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.138 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.153 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.169 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.186 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.253 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.259 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.274 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.284 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.294 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.307 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.316 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.323 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.329 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51.336 UTC [556] WARNING: there is no transaction in progress
2026-08-08 02:26:51,969 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 02:26:51,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 02:26:52,255 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 02:26:52,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 02:26:52,529 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 02:26:52,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 02:26:52,784 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 02:26:52,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 02:26:53,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 02:26:53,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pgkr.c.gcp-upstream.internal';"
2026-08-08 02:26:53,583 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 02:26:53,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 02:26:53,835 [I] DB_PATCHES have already been applied
I20260808 02:26:53.842337 20380 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOT6xt1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 02:26:53.842432 20380 mini_ranger.cc:193] Using host: 127.19.231.20
I20260808 02:26:53.863551 20380 mini_ranger.cc:227] Ranger admin URL: http://127.19.231.20:33115
Aug 08, 2026 2:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 2:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 2:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 2:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 2:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 02:26:55.047086 20380 mini_ranger.cc:160] Time spent starting Ranger: real 8.860s user 0.006s sys 0.057s
Aug 08, 2026 2:27:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 2:27:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 2:27:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 2:27:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 2:27:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 02:27:18.459746 20380 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.231.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.231.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/rangerkms_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.231.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/rangerkms_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.231.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.231.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.231.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/HTTP_127.19.231.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 02:27:18 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156038, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 02:27:18.710456 20380 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 02:27:18.738590 20380 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 02:27:19.542833 20380 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 02:27:19.542990 20380 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms
I20260808 02:27:19.543048 20380 mini_ranger_kms.cc:227] Using postgres at 127.19.231.20:55461
2026-08-08 02:27:19,944 [I] DB FLAVOR :POSTGRES
2026-08-08 02:27:19,944 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 02:27:19,944 [I] Checking connection
2026-08-08 02:27:19,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 02:27:20,191 [I] connection success
2026-08-08 02:27:20,191 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 02:27:20,191 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 02:27:20,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 02:27:20,451 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 02:27:20,451 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 02:27:20,451 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 02:27:20,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.231.20:55461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 02:27:20,712 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 02:27:21.215446 20380 mini_ranger_kms.cc:310] Created kms service
I20260808 02:27:21.433214 20380 mini_ranger_kms.cc:326] Created kudu user
I20260808 02:27:21.485666 20380 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 02:27:21 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156038, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:21.897393 20380 mini_ranger_kms.cc:384] Added ranger policy
I20260808 02:27:21.897758 20380 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOT6xt1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOT6xt1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 02:27:21.897802 20380 mini_ranger_kms.cc:240] Using host: 127.19.231.20
I20260808 02:27:21.919464 20380 mini_ranger_kms.cc:276] Ranger KMS PID: 743
I20260808 02:27:21.919610 20380 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.231.20:59787
Aug 08, 2026 2:27:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 2:27:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 2:27:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 2:27:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 2:27:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 2:27:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 02:27:24 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786156044, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.231.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:24.352705 20380 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.642s user 0.084s sys 0.351s
I20260808 02:27:24.352988 20380 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 02:27:24.353192 20380 mini_ranger_kms.cc:401] 127.19.231.20:59787/kms/v1/keys
Aug 08 02:27:24 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156044, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.231.20@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
Aug 08 02:27:25 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156044, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.231.20@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
Aug 08 02:27:27 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:27:27.479044 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43009
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43009
--ranger_config_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:27.855175 793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:27.855530 793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:27.855686 793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:27.863847 793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:27.864018 793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:27.864128 793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:27.864233 793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:27.864315 793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:27.875211 793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.231.20:59787/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43009
--ranger_config_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:27.876745 793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:27.878428 793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:27.893222 798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:27.893253 799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:27.893991 801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:27.895208 793 server_base.cc:1034] running on GCE node
I20260808 02:27:27.896142 793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:27.897614 793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:27.898882 793 hybrid_clock.cc:648] HybridClock initialized: now 1786156047898822 us; error 46 us; skew 500 ppm
I20260808 02:27:27.899329 793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:27 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156047, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:27.902904 793 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:27:27.904652 793 webserver.cc:460] Webserver started at http://127.19.231.62:38287/ using document root <none> and password file <none>
I20260808 02:27:27.905413 793 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:27.905544 793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:27.905907 793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:27:27 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156047, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:27.969370 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "1adc42c7c4b64981a25cbf1c79e861a0"
format_stamp: "Formatted at 2026-08-08 02:27:27 on dist-test-slave-pgkr"
server_key: "6baafdbad0c23470be893aa7331a22c3"
server_key_iv: "93d330e31409e1a9e9c0670ac4fa43e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:27.970329 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "1adc42c7c4b64981a25cbf1c79e861a0"
format_stamp: "Formatted at 2026-08-08 02:27:27 on dist-test-slave-pgkr"
server_key: "6baafdbad0c23470be893aa7331a22c3"
server_key_iv: "93d330e31409e1a9e9c0670ac4fa43e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:27.976339 793 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 02:27:27.994580 808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:27.996397 793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:27:27.996634 793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "1adc42c7c4b64981a25cbf1c79e861a0"
format_stamp: "Formatted at 2026-08-08 02:27:27 on dist-test-slave-pgkr"
server_key: "6baafdbad0c23470be893aa7331a22c3"
server_key_iv: "93d330e31409e1a9e9c0670ac4fa43e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:27.996896 793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:28.031018 793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:28.035759 793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:28.036150 793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:28.057402 793 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43009
I20260808 02:27:28.057427 862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43009 every 8 connection(s)
I20260808 02:27:28.058919 793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:27:28.062863 863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:28.067178 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 793
I20260808 02:27:28.067423 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:27:28.068321 20380 external_mini_cluster.cc:1428] Setting key 4180d790fae81e5a94a3108d193008e9
I20260808 02:27:28.075289 863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Bootstrap starting.
I20260808 02:27:28.078975 863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:28.080155 863 log.cc:824] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:28.083729 863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: No bootstrap required, opened a new log
Aug 08 02:27:28 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:28.092141 863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:28.092669 863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:28.092796 863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adc42c7c4b64981a25cbf1c79e861a0, State: Initialized, Role: FOLLOWER
I20260808 02:27:28.093509 863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:28.093753 863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:27:28.093906 863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:27:28.093937 867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:28.070750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.19.231.62:43009)
0808 02:27:28.071619 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:28.071632 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:28.071687 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:27:28.073588 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:28.073613 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:28.074014 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:28.074359 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:28.076129 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:28.077470 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:28.079433 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:28.080131 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:28.080265 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:28.085452 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:28.085463 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:28.085477 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:28.085517 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:28.089007 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:28.089856 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:28.089862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:28.089869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:28.090037 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:28.090521 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:28.090525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:28.090527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:28.091217 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:28.091577 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:28.092065 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 02:27:28.092532 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":429,"threads_started":1}
I20260808 02:27:28.094113 863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:28.096562 863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:28.097144 863 leader_election.cc:302] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1adc42c7c4b64981a25cbf1c79e861a0; no voters:
I20260808 02:27:28.097747 863 leader_election.cc:288] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:27:28.097934 869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:27:28.098752 869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1adc42c7c4b64981a25cbf1c79e861a0, State: Running, Role: LEADER
I20260808 02:27:28.099419 869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:28.099854 863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:28.102772 870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } } }
I20260808 02:27:28.103202 870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:28.103724 871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1adc42c7c4b64981a25cbf1c79e861a0. Latest consensus state: current_term: 1 leader_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } } }
I20260808 02:27:28.104091 871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:28.105072 877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:27:28.109716 877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:27:28.115340 863 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:27:28.128230 877 catalog_manager.cc:1347] Generated new cluster ID: f9d70e61ff5b433991afd5882a0f7420
I20260808 02:27:28.128489 877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:27:28.143401 877 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:27:28.145671 877 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:27:28.157600 877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Generated new TSK 0
I20260808 02:27:28.158842 877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:27:28.984665 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43009
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:27:29 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786156049, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 02:27:29.435001 911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:29.435364 911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:29.435565 911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:29.443351 911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:29.443516 911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:29.443754 911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:27:29.455188 911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.231.20:59787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:29.456713 911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:29.458204 911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:29.476979 922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:29.484419 923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:29.484676 925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:29.485661 911 server_base.cc:1034] running on GCE node
I20260808 02:27:29.486749 911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:29.487923 911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:29.489214 911 hybrid_clock.cc:648] HybridClock initialized: now 1786156049489135 us; error 68 us; skew 500 ppm
I20260808 02:27:29.489590 911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:29 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156049, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:29.493237 911 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:27:29.499020 911 webserver.cc:460] Webserver started at http://127.19.231.1:40051/ using document root <none> and password file <none>
I20260808 02:27:29.499946 911 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:29.500119 911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:29.500542 911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:27:29 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156049, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:29.524622 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "7bae0ad0b25d49c59f90f6af2e61d91a"
format_stamp: "Formatted at 2026-08-08 02:27:29 on dist-test-slave-pgkr"
server_key: "74bdb3725748c882f45e2cb33d303154"
server_key_iv: "c20a90c68fc63adc68810bfecf324603"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:29.526175 911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "7bae0ad0b25d49c59f90f6af2e61d91a"
format_stamp: "Formatted at 2026-08-08 02:27:29 on dist-test-slave-pgkr"
server_key: "74bdb3725748c882f45e2cb33d303154"
server_key_iv: "c20a90c68fc63adc68810bfecf324603"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:29.544049 911 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.011s
I20260808 02:27:29.578719 933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:29.580915 911 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.008s sys 0.002s
I20260808 02:27:29.581275 911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "7bae0ad0b25d49c59f90f6af2e61d91a"
format_stamp: "Formatted at 2026-08-08 02:27:29 on dist-test-slave-pgkr"
server_key: "74bdb3725748c882f45e2cb33d303154"
server_key_iv: "c20a90c68fc63adc68810bfecf324603"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:29.581745 911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:29.646139 911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:29.652281 911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:29.652647 911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:29.654080 911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:29.656208 911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:29.656323 911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:29.656486 911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:29.656562 911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:29.750066 911 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:34453
I20260808 02:27:29.750128 1047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:34453 every 8 connection(s)
I20260808 02:27:29.752065 911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:27:29.760941 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 911
I20260808 02:27:29.761181 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:27:29.762076 20380 external_mini_cluster.cc:1428] Setting key 5e9799587d62e2a8de740699171a1b7e
Aug 08 02:27:29 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156049, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:29.789367 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:29.755561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:50715 (local address 127.19.231.62:43009)
0808 02:27:29.767096 (+ 11535us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:29.767106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:29.769715 (+ 2609us) server_negotiation.cc:407] Connection header received
0808 02:27:29.769773 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:29.769776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:29.769851 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:29.769979 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:29.772029 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:29.773189 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:29.775723 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:29.776407 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:29.776533 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:29.781553 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:29.781561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:29.781567 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:29.781611 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:29.784833 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:29.785827 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:29.785834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:29.785837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:29.786010 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:29.786670 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:29.786676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:29.786679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:29.787081 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:29.787236 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:29.788505 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0808 02:27:29.788835 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11369,"thread_start_us":11246,"threads_started":1}
I20260808 02:27:29.792346 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:29.756228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43009 (local address 127.19.231.1:50715)
0808 02:27:29.767507 (+ 11279us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:29.767586 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:29.769390 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:29.770282 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:29.770295 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:29.770978 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:29.771811 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:29.771835 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:29.773388 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:29.773393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:29.775545 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:29.775560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:29.776840 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:29.776848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:29.777160 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:29.778082 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:29.778128 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:29.781241 (+ 3113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:29.784994 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:29.785003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:29.785021 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:29.785664 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:29.786194 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:29.786200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:29.786204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:29.786466 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:29.787361 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:29.787369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:29.787946 (+ 577us) client_negotiation.cc:769] Sending connection context
0808 02:27:29.790156 (+ 2210us) client_negotiation.cc:240] Negotiation successful
0808 02:27:29.790773 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10965,"thread_start_us":10746,"threads_started":1}
I20260808 02:27:29.794700 1048 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43009
I20260808 02:27:29.795089 1048 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:29.796103 1048 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
I20260808 02:27:29.798976 827 ts_manager.cc:194] Registered new tserver with Master: 7bae0ad0b25d49c59f90f6af2e61d91a (127.19.231.1:34453)
I20260808 02:27:29.801134 827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:50715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:27:29.843407 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43009
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:27:29 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156049, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
Aug 08 02:27:30 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156049, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
W20260808 02:27:30.390581 1054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:30.391114 1054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:30.391319 1054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:30.403071 1054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:30.403293 1054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:30.403532 1054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:27:30.416812 1054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.231.20:59787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:30.418695 1054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:30.420331 1054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:30.467878 1063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:30.476428 1062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:30.476872 1054 server_base.cc:1034] running on GCE node
W20260808 02:27:30.477825 1066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:30.478767 1054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:30.483131 1054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:30.484441 1054 hybrid_clock.cc:648] HybridClock initialized: now 1786156050484367 us; error 62 us; skew 500 ppm
I20260808 02:27:30.484872 1054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:30 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156050, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:30.489230 1054 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:27:30.499414 1054 webserver.cc:460] Webserver started at http://127.19.231.2:44065/ using document root <none> and password file <none>
I20260808 02:27:30.500254 1054 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:30.500411 1054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:30.500823 1054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:27:30 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156050, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:30.518018 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "ddcd7ae5001847c09d194513bbf43233"
format_stamp: "Formatted at 2026-08-08 02:27:30 on dist-test-slave-pgkr"
server_key: "9f1370e91a0b78fa1030c52216740d9e"
server_key_iv: "d705fca085b522cdf0986314b0cf2754"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:30.518998 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "ddcd7ae5001847c09d194513bbf43233"
format_stamp: "Formatted at 2026-08-08 02:27:30 on dist-test-slave-pgkr"
server_key: "9f1370e91a0b78fa1030c52216740d9e"
server_key_iv: "d705fca085b522cdf0986314b0cf2754"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:30.525406 1054 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260808 02:27:30.551698 1073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:30.553992 1054 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:27:30.554222 1054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "ddcd7ae5001847c09d194513bbf43233"
format_stamp: "Formatted at 2026-08-08 02:27:30 on dist-test-slave-pgkr"
server_key: "9f1370e91a0b78fa1030c52216740d9e"
server_key_iv: "d705fca085b522cdf0986314b0cf2754"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:30.554534 1054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:30.596210 1054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:30.601902 1054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:30.602262 1054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:30.603721 1054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:30.605898 1054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:30.606000 1054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:30.606184 1054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:30.606278 1054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:30.648983 1054 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:33705
I20260808 02:27:30.649013 1197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:33705 every 8 connection(s)
I20260808 02:27:30.650851 1054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:27:30.655568 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 1054
I20260808 02:27:30.655814 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:27:30.656489 20380 external_mini_cluster.cc:1428] Setting key b5395ac3302152d03a1aef083c5e27b4
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Aug 08 02:27:30 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156050, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:30.703537 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:30.655004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:40449 (local address 127.19.231.62:43009)
0808 02:27:30.663021 (+ 8017us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:30.663029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:30.671486 (+ 8457us) server_negotiation.cc:407] Connection header received
0808 02:27:30.674497 (+ 3011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:30.674502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:30.674642 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:30.674755 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:30.685229 (+ 10474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:30.686690 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:30.689801 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:30.690448 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:30.690533 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:30.695168 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:30.695178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:30.695184 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:30.695233 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:30.699186 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:30.700198 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:30.700206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:30.700213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:30.700370 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:30.700931 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:30.700938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:30.700943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:30.701360 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:30.701506 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:30.702580 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0808 02:27:30.702888 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7852,"thread_start_us":15626,"threads_started":1}
I20260808 02:27:30.706496 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:30.654709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43009 (local address 127.19.231.2:40449)
0808 02:27:30.671283 (+ 16574us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:30.671345 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:30.674160 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:30.683053 (+ 8893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:30.683071 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:30.683841 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:30.684954 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:30.684989 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:30.686941 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:30.686949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:30.689588 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:30.689615 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:30.690962 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:30.690972 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:30.691284 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:30.692094 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:30.692119 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:30.694850 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:30.699370 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:30.699382 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:30.699403 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:30.700039 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:30.700521 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:30.700528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:30.700534 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:30.700792 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:30.701624 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:30.701635 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:30.702119 (+ 484us) client_negotiation.cc:769] Sending connection context
0808 02:27:30.704138 (+ 2019us) client_negotiation.cc:240] Negotiation successful
0808 02:27:30.704740 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16207,"thread_start_us":15953,"threads_started":1}
I20260808 02:27:30.708657 1198 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43009
I20260808 02:27:30.708993 1198 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:30.709865 1198 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
I20260808 02:27:30.711561 827 ts_manager.cc:194] Registered new tserver with Master: ddcd7ae5001847c09d194513bbf43233 (127.19.231.2:33705)
I20260808 02:27:30.712864 827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:40449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:27:30.734321 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43009
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:27:30.808449 1048 heartbeater.cc:503] Master 127.19.231.62:43009 was elected leader, sending a full tablet report...
W20260808 02:27:31.115180 1204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:31.115520 1204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:31.115690 1204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:31.123090 1204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:31.123250 1204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:31.123420 1204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:27:31.133610 1204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.231.20:59787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:31.135048 1204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:31.136431 1204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:31.149998 1210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:31.150035 1211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:31.150070 1213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:31.150933 1204 server_base.cc:1034] running on GCE node
I20260808 02:27:31.151571 1204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:31.152580 1204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:31.153820 1204 hybrid_clock.cc:648] HybridClock initialized: now 1786156051153774 us; error 57 us; skew 500 ppm
I20260808 02:27:31.154161 1204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:31 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156051, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:31.157546 1204 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:27:31.159243 1204 webserver.cc:460] Webserver started at http://127.19.231.3:34723/ using document root <none> and password file <none>
I20260808 02:27:31.159958 1204 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:31.160117 1204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:31.160444 1204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 02:27:31 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156051, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:31.178041 1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "6c468be546594993a9c0bd1e66d3f513"
format_stamp: "Formatted at 2026-08-08 02:27:31 on dist-test-slave-pgkr"
server_key: "54c30b5c36ebd892695b9a3823b1c5e6"
server_key_iv: "018837568c9fe560b8a5591b2f9e7ea4"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:31.179066 1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "6c468be546594993a9c0bd1e66d3f513"
format_stamp: "Formatted at 2026-08-08 02:27:31 on dist-test-slave-pgkr"
server_key: "54c30b5c36ebd892695b9a3823b1c5e6"
server_key_iv: "018837568c9fe560b8a5591b2f9e7ea4"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:31.185309 1204 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 02:27:31.203095 1220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:31.204929 1204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:27:31.205137 1204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "6c468be546594993a9c0bd1e66d3f513"
format_stamp: "Formatted at 2026-08-08 02:27:31 on dist-test-slave-pgkr"
server_key: "54c30b5c36ebd892695b9a3823b1c5e6"
server_key_iv: "018837568c9fe560b8a5591b2f9e7ea4"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:31.205428 1204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:31.220788 1204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:31.226006 1204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:31.226370 1204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:31.229229 1204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:31.231133 1204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:31.231227 1204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:31.231353 1204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:31.231418 1204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:31.265564 1204 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:44259
I20260808 02:27:31.265596 1337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:44259 every 8 connection(s)
I20260808 02:27:31.266899 1204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:27:31.270264 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 1204
I20260808 02:27:31.270468 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:27:31.271039 20380 external_mini_cluster.cc:1428] Setting key 7ee921761cc1f2b84371b012099befcc
Aug 08 02:27:31 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156051, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:31.287251 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:31.269571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:50479 (local address 127.19.231.62:43009)
0808 02:27:31.270179 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:31.270186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:31.271385 (+ 1199us) server_negotiation.cc:407] Connection header received
0808 02:27:31.273581 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:31.273586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:31.273671 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:31.273813 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:31.275402 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.276101 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.277609 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.278049 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.278142 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:31.281361 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:31.281365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:31.281369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:31.281400 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:31.284011 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.284712 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.284718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.284739 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.284876 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.285363 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.285368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.285371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.285748 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:31.285897 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:31.286819 (+ 922us) server_negotiation.cc:299] Negotiation successful
0808 02:27:31.287004 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":255,"threads_started":1}
I20260808 02:27:31.288412 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:31.270276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43009 (local address 127.19.231.3:50479)
0808 02:27:31.271206 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:31.271255 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:31.273322 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:31.274060 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:31.274073 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:31.274532 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:31.275196 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:31.275215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.276244 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.276248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:31.277480 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:31.277492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.278240 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.278245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:31.278448 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:31.279119 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:31.279141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:31.281147 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:31.284134 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:31.284140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:31.284160 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:31.284582 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:31.284996 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:31.285001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:31.285005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:31.285237 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:31.285894 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:31.285901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:31.286304 (+ 403us) client_negotiation.cc:769] Sending connection context
0808 02:27:31.286682 (+ 378us) client_negotiation.cc:240] Negotiation successful
0808 02:27:31.287105 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":416,"threads_started":1}
I20260808 02:27:31.289809 1338 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43009
I20260808 02:27:31.290102 1338 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:31.290860 1338 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
I20260808 02:27:31.292205 827 ts_manager.cc:194] Registered new tserver with Master: 6c468be546594993a9c0bd1e66d3f513 (127.19.231.3:44259)
I20260808 02:27:31.293035 827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:50479
I20260808 02:27:31.299085 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:27:31.300940 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 911
Aug 08 02:27:31 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:27:31.318022 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:31.301752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.19.231.1:34453)
0808 02:27:31.302463 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:31.302470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:31.302495 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:27:31.302550 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:31.302556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:31.302854 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:31.302983 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:31.304344 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.305350 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.307010 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.307560 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.307664 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:31.312317 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:31.312330 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:31.312343 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:31.312382 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:31.315144 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.315861 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.315869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.315872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.316018 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.316408 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.316414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.316419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.316774 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:31.317091 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:31.317377 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 02:27:31.317653 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":390,"threads_started":1}
I20260808 02:27:31.319324 1043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:31.404284 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 911
I20260808 02:27:31.424876 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 1054
Aug 08 02:27:31 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:27:31.442142 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:31.426014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.19.231.2:33705)
0808 02:27:31.426539 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:31.426546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:31.426571 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:27:31.426684 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:31.426695 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:31.426962 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:31.427135 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:31.428475 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.429458 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.430921 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.431490 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.431638 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:31.436237 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:31.436248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:31.436263 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:31.436308 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:31.439161 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.439918 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.439926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.439928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.440076 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.440461 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.440467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.440472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.440841 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:31.441161 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:31.441492 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 02:27:31.441796 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
I20260808 02:27:31.443526 1192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:31.529085 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 1054
I20260808 02:27:31.554075 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 1204
Aug 08 02:27:31 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:27:31.573076 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:31.555209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.19.231.3:44259)
0808 02:27:31.555962 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:31.555971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:31.556009 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:27:31.556087 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:31.556095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:31.556337 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:31.556556 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:31.558415 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.559822 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.561451 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:31.562161 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:31.562344 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:31.567229 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:31.567243 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:31.567257 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:31.567298 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:31.570103 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.570852 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.570859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.570861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.570997 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:31.571389 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:31.571394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:31.571397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:31.571766 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:31.572123 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:31.572425 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:27:31.572706 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":389,"threads_started":1}
I20260808 02:27:31.574509 1333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:31.660881 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 1204
I20260808 02:27:31.680151 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 793
I20260808 02:27:31.681489 858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:31.790678 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 793
I20260808 02:27:31.818850 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43009
--webserver_interface=127.19.231.62
--webserver_port=38287
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43009
--ranger_config_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:32.161792 1352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:32.162159 1352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:32.162325 1352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:32.169747 1352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:32.169909 1352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:32.170028 1352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:32.170138 1352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:32.170234 1352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:32.180591 1352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43009
--ranger_config_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=38287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:32.181960 1352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:32.183418 1352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:32.195444 1357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:32.195498 1358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:32.195533 1360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:32.196663 1352 server_base.cc:1034] running on GCE node
I20260808 02:27:32.197455 1352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:32.198925 1352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:32.200167 1352 hybrid_clock.cc:648] HybridClock initialized: now 1786156052200117 us; error 44 us; skew 500 ppm
I20260808 02:27:32.200538 1352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:32 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156052, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:32.203964 1352 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:27:32.205480 1352 webserver.cc:460] Webserver started at http://127.19.231.62:38287/ using document root <none> and password file <none>
I20260808 02:27:32.206187 1352 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:32.206331 1352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:27:32 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156052, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:32.258010 1352 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:27:32.261227 1368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:32.262775 1352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:27:32.262970 1352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "1adc42c7c4b64981a25cbf1c79e861a0"
format_stamp: "Formatted at 2026-08-08 02:27:27 on dist-test-slave-pgkr"
server_key: "6baafdbad0c23470be893aa7331a22c3"
server_key_iv: "93d330e31409e1a9e9c0670ac4fa43e5"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:32.263952 1352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:32.292577 1352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:32.296680 1352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:32.296972 1352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:32.316336 1352 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43009
I20260808 02:27:32.316372 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43009 every 8 connection(s)
I20260808 02:27:32.317844 1352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:27:32.318398 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 1352
I20260808 02:27:32.320354 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:34453
--local_ip_for_outbound_sockets=127.19.231.1
--tserver_master_addrs=127.19.231.62:43009
--webserver_port=40051
--webserver_interface=127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:27:32.325557 1423 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:27:32.328879 1423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Bootstrap starting.
I20260808 02:27:32.353734 1423 log.cc:824] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:32.363494 1423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:27:32.364073 1423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Bootstrap complete.
I20260808 02:27:32.373618 1423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:32.374522 1423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adc42c7c4b64981a25cbf1c79e861a0, State: Initialized, Role: FOLLOWER
I20260808 02:27:32.375157 1423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:32.375406 1423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:27:32.375536 1423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:27:32.375721 1423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:27:32.379748 1423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:32.380285 1423 leader_election.cc:302] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1adc42c7c4b64981a25cbf1c79e861a0; no voters:
I20260808 02:27:32.381012 1423 leader_election.cc:288] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:27:32.381176 1427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:27:32.382305 1427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [term 2 LEADER]: Becoming Leader. State: Replica: 1adc42c7c4b64981a25cbf1c79e861a0, State: Running, Role: LEADER
I20260808 02:27:32.383010 1427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } }
I20260808 02:27:32.383370 1423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:32.385890 1429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1adc42c7c4b64981a25cbf1c79e861a0. Latest consensus state: current_term: 2 leader_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } } }
I20260808 02:27:32.386366 1429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:32.387277 1428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc42c7c4b64981a25cbf1c79e861a0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43009 } } }
I20260808 02:27:32.387634 1428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:32.400295 1433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:27:32.407485 1433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:27:32.409729 1433 catalog_manager.cc:1259] Loaded cluster ID: f9d70e61ff5b433991afd5882a0f7420
I20260808 02:27:32.409993 1433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:27:32.415763 1433 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:27:32.417790 1423 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 02:27:32.418180 1433 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Loaded TSK: 0
I20260808 02:27:32.419934 1433 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:27:32.758858 1425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:32.759351 1425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:32.759588 1425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:32.771904 1425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:32.772161 1425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:32.772444 1425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:27:32.788844 1425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:34453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=40051
--tserver_master_addrs=127.19.231.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:32.790699 1425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:32.792619 1425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:32.815904 1460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:32.827054 1461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:32.833117 1425 server_base.cc:1034] running on GCE node
W20260808 02:27:32.834084 1464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:32.835165 1425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:32.846931 1425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:32.848206 1425 hybrid_clock.cc:648] HybridClock initialized: now 1786156052848120 us; error 93 us; skew 500 ppm
I20260808 02:27:32.848784 1425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:32 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156052, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:32.853987 1425 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:27:32.860448 1425 webserver.cc:460] Webserver started at http://127.19.231.1:40051/ using document root <none> and password file <none>
I20260808 02:27:32.861634 1425 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:32.861835 1425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:27:32 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156052, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:32.896199 1425 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 02:27:32.906586 1473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:32.908946 1425 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.010s sys 0.001s
I20260808 02:27:32.909191 1425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "7bae0ad0b25d49c59f90f6af2e61d91a"
format_stamp: "Formatted at 2026-08-08 02:27:29 on dist-test-slave-pgkr"
server_key: "74bdb3725748c882f45e2cb33d303154"
server_key_iv: "c20a90c68fc63adc68810bfecf324603"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:32.910976 1425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:32.933681 1425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:32.939841 1425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:32.940277 1425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:32.951656 1425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:32.963312 1425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:32.963474 1425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:32.963637 1425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:32.963737 1425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:33.055831 1425 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:34453
I20260808 02:27:33.055893 1592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:34453 every 8 connection(s)
I20260808 02:27:33.058290 1425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:27:33.064585 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 1425
I20260808 02:27:33.065598 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:33705
--local_ip_for_outbound_sockets=127.19.231.2
--tserver_master_addrs=127.19.231.62:43009
--webserver_port=44065
--webserver_interface=127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:27:33 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156052, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:33.119513 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:33.064846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43009 (local address 127.19.231.1:37899)
0808 02:27:33.090158 (+ 25312us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:33.090228 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:33.093122 (+ 2894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:33.095461 (+ 2339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:33.095475 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:33.096221 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:33.097034 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:33.097062 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.099636 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.099645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:33.102066 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:33.102095 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.103693 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.103706 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:33.104037 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:33.104887 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:33.104928 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:33.108191 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:33.113312 (+ 5121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:33.113322 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:33.113340 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:33.114048 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:33.114848 (+ 800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:33.114854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:33.114858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:33.115169 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:33.116421 (+ 1252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:33.116430 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:33.116877 (+ 447us) client_negotiation.cc:769] Sending connection context
0808 02:27:33.117273 (+ 396us) client_negotiation.cc:240] Negotiation successful
0808 02:27:33.117883 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24873,"thread_start_us":24896,"threads_started":1}
I20260808 02:27:33.122423 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:33.064339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:37899 (local address 127.19.231.62:43009)
0808 02:27:33.089741 (+ 25402us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:33.089766 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:33.090434 (+ 668us) server_negotiation.cc:407] Connection header received
0808 02:27:33.094139 (+ 3705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:33.094166 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:33.094793 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:33.095284 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:33.097315 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.099440 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.102247 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.103361 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.103602 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:33.108507 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:33.108519 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:33.108540 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:33.108615 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:33.113089 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:33.114256 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:33.114266 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:33.114275 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:33.114679 (+ 404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:33.115328 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:33.115334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:33.115339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:33.116075 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:33.116307 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:33.120331 (+ 4024us) server_negotiation.cc:299] Negotiation successful
0808 02:27:33.120788 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25051,"thread_start_us":24798,"threads_started":1}
I20260808 02:27:33.127221 1593 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43009
I20260808 02:27:33.127609 1593 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:33.128614 1593 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
W20260808 02:27:33.132316 1593 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:43009 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:27:33.132575 1593 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:33.132865 1593 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
I20260808 02:27:33.134227 1593 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:33.134490 1593 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
W20260808 02:27:33.135705 1593 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:27:33.515887 1598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:33.516328 1598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:33.516517 1598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:33.524631 1598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:33.524808 1598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:33.525023 1598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:27:33.535799 1598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:33705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=44065
--tserver_master_addrs=127.19.231.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:33.537256 1598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:33.538736 1598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:33.553619 1612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:33.554114 1614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:33.553891 1611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:33.555043 1598 server_base.cc:1034] running on GCE node
I20260808 02:27:33.555650 1598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:33.556655 1598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:33.557911 1598 hybrid_clock.cc:648] HybridClock initialized: now 1786156053557834 us; error 61 us; skew 500 ppm
I20260808 02:27:33.558249 1598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:33 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156053, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:33.561826 1598 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:27:33.563473 1598 webserver.cc:460] Webserver started at http://127.19.231.2:44065/ using document root <none> and password file <none>
I20260808 02:27:33.564203 1598 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:33.564366 1598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:27:33 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156053, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:33.591660 1598 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 02:27:33.595096 1624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:33.596786 1598 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 02:27:33.596980 1598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "ddcd7ae5001847c09d194513bbf43233"
format_stamp: "Formatted at 2026-08-08 02:27:30 on dist-test-slave-pgkr"
server_key: "9f1370e91a0b78fa1030c52216740d9e"
server_key_iv: "d705fca085b522cdf0986314b0cf2754"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:33.598052 1598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:33.616895 1598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:33.621364 1598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:33.621659 1598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:33.623236 1598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:33.625231 1598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:33.625366 1598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:33.625528 1598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:33.625638 1598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:33.727715 1598 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:33705
I20260808 02:27:33.727897 1743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:33705 every 8 connection(s)
I20260808 02:27:33.729856 1598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:27:33.738463 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 1598
I20260808 02:27:33.739409 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.231.20:59787/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:44259
--local_ip_for_outbound_sockets=127.19.231.3
--tserver_master_addrs=127.19.231.62:43009
--webserver_port=34723
--webserver_interface=127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:27:33 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156053, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:33.788395 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:33.734322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:41227 (local address 127.19.231.62:43009)
0808 02:27:33.757242 (+ 22920us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:33.757251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:33.760967 (+ 3716us) server_negotiation.cc:407] Connection header received
0808 02:27:33.761046 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:33.761052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:33.761142 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:33.761250 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:33.763744 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.764958 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.767557 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.768221 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.768332 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:33.778828 (+ 10496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:33.778837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:33.778844 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:33.778901 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:33.783425 (+ 4524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:33.784700 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:33.784708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:33.784713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:33.784916 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:33.785664 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:33.785670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:33.785675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:33.786165 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:33.786338 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:33.787626 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0808 02:27:33.787896 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22765,"thread_start_us":22613,"threads_started":1}
I20260808 02:27:33.789752 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:33.733639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43009 (local address 127.19.231.2:41227)
0808 02:27:33.757903 (+ 24264us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:33.757966 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:33.760502 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:33.761762 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:33.761775 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:33.762481 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:33.763445 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:33.763474 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.765199 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.765204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:33.767339 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:33.767355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:33.768793 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:33.768801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:33.769105 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:33.774824 (+ 5719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:33.774860 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:33.778424 (+ 3564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:33.783692 (+ 5268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:33.783703 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:33.783725 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:33.784474 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:33.785122 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:33.785130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:33.785136 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:33.785469 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:33.786461 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:33.786471 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:33.787061 (+ 590us) client_negotiation.cc:769] Sending connection context
0808 02:27:33.787483 (+ 422us) client_negotiation.cc:240] Negotiation successful
0808 02:27:33.788165 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23907,"thread_start_us":23653,"threads_started":1}
I20260808 02:27:33.791594 1744 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43009
I20260808 02:27:33.791986 1744 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:33.792935 1744 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
I20260808 02:27:33.795238 1387 ts_manager.cc:194] Registered new tserver with Master: ddcd7ae5001847c09d194513bbf43233 (127.19.231.2:33705)
I20260808 02:27:33.797833 1387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:41227
Aug 08 02:27:33 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786156053, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:34.135860 1593 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:34.136355 1593 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
I20260808 02:27:34.137812 1387 ts_manager.cc:194] Registered new tserver with Master: 7bae0ad0b25d49c59f90f6af2e61d91a (127.19.231.1:34453)
I20260808 02:27:34.139123 1387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:37899
W20260808 02:27:34.227807 1748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:34.228276 1748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:34.228461 1748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:34.241467 1748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:34.241710 1748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:34.241971 1748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
Aug 08 02:27:34 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156053, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:34.259158 1748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.231.20:59787/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:44259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=34723
--tserver_master_addrs=127.19.231.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:34.260941 1748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:34.262863 1748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:34.332865 1755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:34.332890 1756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:34.343506 1748 server_base.cc:1034] running on GCE node
W20260808 02:27:34.344208 1758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:34.345155 1748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:34.350419 1748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:34.351912 1748 hybrid_clock.cc:648] HybridClock initialized: now 1786156054351798 us; error 114 us; skew 500 ppm
I20260808 02:27:34.352459 1748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:34 dist-test-slave-pgkr krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156054, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:34.357290 1748 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:27:34.385625 1748 webserver.cc:460] Webserver started at http://127.19.231.3:34723/ using document root <none> and password file <none>
I20260808 02:27:34.386951 1748 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:34.387157 1748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 02:27:34 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156054, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:34.435778 1748 fs_manager.cc:709] Time spent opening directory manager: real 0.026s user 0.017s sys 0.004s
I20260808 02:27:34.451398 1767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:34.453833 1748 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.009s
I20260808 02:27:34.464397 1748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "6c468be546594993a9c0bd1e66d3f513"
format_stamp: "Formatted at 2026-08-08 02:27:31 on dist-test-slave-pgkr"
server_key: "54c30b5c36ebd892695b9a3823b1c5e6"
server_key_iv: "018837568c9fe560b8a5591b2f9e7ea4"
server_key_version: "kuduclusterkey@0"
I20260808 02:27:34.466135 1748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:34.543360 1748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:34.549685 1748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:34.550130 1748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:34.563807 1748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:34.576092 1748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:34.576321 1748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 02:27:34.576493 1748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:34.576630 1748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 02:27:34 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156053, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for HTTP/127.19.231.20@KRBTEST.COM
I20260808 02:27:34.803460 1744 heartbeater.cc:503] Master 127.19.231.62:43009 was elected leader, sending a full tablet report...
I20260808 02:27:34.835506 1748 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:44259
I20260808 02:27:34.835602 1887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:44259 every 8 connection(s)
I20260808 02:27:34.837500 1748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:27:34.842787 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 1748
I20260808 02:27:34.877691 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.844174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.19.231.62:43009)
0808 02:27:34.856119 (+ 11945us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:34.856143 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:34.856171 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:27:34.856286 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:34.856295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:34.856405 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:34.856535 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:34.859472 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.860931 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.863617 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.864529 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.864646 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:34.868792 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:34.868802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:34.868807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:34.868846 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:34.873492 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:34.874405 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:34.874411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:34.874414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:34.874590 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:34.875127 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:34.875132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:34.875136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:34.875495 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:34.875793 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:34.877061 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0808 02:27:34.877319 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10292,"thread_start_us":14442,"threads_started":1}
Aug 08 02:27:34 dist-test-slave-pgkr krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156054, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:34.893991 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.854728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43009 (local address 127.19.231.3:44011)
0808 02:27:34.866851 (+ 12123us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:34.866912 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:34.869604 (+ 2692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:34.874036 (+ 4432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:34.874049 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:34.874796 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:34.875617 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:34.875643 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.877428 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.877434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:34.879602 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:34.879618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.881090 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.881099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:34.881394 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:34.882193 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:34.882225 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:34.885110 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:34.889144 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:34.889154 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:34.889171 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:34.889750 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:34.890195 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:34.890200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:34.890204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:34.890462 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:34.891214 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:34.891223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:34.891643 (+ 420us) client_negotiation.cc:769] Sending connection context
0808 02:27:34.891984 (+ 341us) client_negotiation.cc:240] Negotiation successful
0808 02:27:34.892563 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11725,"thread_start_us":15203,"threads_started":1}
I20260808 02:27:34.895442 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.853862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:44011 (local address 127.19.231.62:43009)
0808 02:27:34.873418 (+ 19556us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:34.873425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:34.873443 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:27:34.873506 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:34.873510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:34.873605 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:34.873709 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:34.875892 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.877222 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.879794 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.880634 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.880759 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:34.885392 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:34.885401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:34.885406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:34.885444 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:34.888940 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:34.889896 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:34.889902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:34.889906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:34.890042 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:34.890604 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:34.890634 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:34.890637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:34.890981 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:34.891140 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:34.894791 (+ 3651us) server_negotiation.cc:299] Negotiation successful
0808 02:27:34.895031 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19397,"thread_start_us":19242,"threads_started":1}
I20260808 02:27:34.897192 1888 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43009
I20260808 02:27:34.897505 1888 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:34.898375 1888 heartbeater.cc:511] Master 127.19.231.62:43009 requested a full tablet report, sending...
I20260808 02:27:34.900192 1387 ts_manager.cc:194] Registered new tserver with Master: 6c468be546594993a9c0bd1e66d3f513 (127.19.231.3:44259)
I20260808 02:27:34.901368 1387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:44011
I20260808 02:27:34.905665 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:27:34.941635 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.910041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.19.231.62:43009)
0808 02:27:34.910213 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:34.910219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:34.923310 (+ 13091us) server_negotiation.cc:407] Connection header received
0808 02:27:34.923548 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:34.923554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:34.923675 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:34.923777 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:34.926570 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.927759 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.929780 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:34.930448 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:34.930591 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:34.934378 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:34.934385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:34.934389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:34.934431 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:34.937963 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:34.938881 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:34.938886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:34.938890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:34.939067 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:34.939849 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:34.939854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:34.939858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:34.940225 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:34.940453 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:34.941039 (+ 586us) server_negotiation.cc:299] Negotiation successful
0808 02:27:34.941274 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:27:34.948379 1387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:27:34.959623 1387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:27:35.023430 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.993200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.19.231.1:34453)
0808 02:27:35.013016 (+ 19816us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.013025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.013056 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:27:35.013167 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.013175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.013445 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.013637 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:35.014631 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.016470 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.019562 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.021181 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.021304 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.021545 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.022883 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.023065 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19528,"thread_start_us":19383,"threads_started":1}
I20260808 02:27:35.023711 1913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.992826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:34453 (local address 127.0.0.1:34990)
0808 02:27:35.012195 (+ 19369us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:35.012214 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:35.012362 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:35.013951 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:35.013959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:35.013991 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:35.014388 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.014401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.016624 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.016630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.019368 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.019384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.021702 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.021708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.021908 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.021957 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 02:27:35.022085 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:27:35.022298 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19229,"mutex_wait_us":27,"thread_start_us":19116,"threads_started":1}
I20260808 02:27:35.025249 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.992170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.19.231.3:44259)
0808 02:27:35.013906 (+ 21736us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.013912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.013940 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:27:35.014031 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.014036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.014226 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.014458 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:35.016371 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.017993 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.021045 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.022784 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.022953 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.023152 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.024654 (+ 1502us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.024855 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21551,"thread_start_us":21411,"threads_started":1}
I20260808 02:27:35.025787 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.991806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:44259 (local address 127.0.0.1:40518)
0808 02:27:35.007620 (+ 15814us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:35.007690 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:35.007934 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:35.014451 (+ 6517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:35.014457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:35.014497 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:35.016119 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.016136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.018313 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.018320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.020832 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.020854 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.024250 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.024261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.024431 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.024466 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:27:35.025251 (+ 785us) client_negotiation.cc:240] Negotiation successful
0808 02:27:35.025398 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15163,"mutex_wait_us":5,"thread_start_us":14566,"threads_started":1}
I20260808 02:27:35.028237 1522 tablet_service.cc:1467] Processing CreateTablet for tablet 98cdedef5e79436fabc52385b4adb25b (DEFAULT_TABLE table=test-table [id=1f6a225a0c9f4f3d8a730234b91ad9ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:35.029327 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.996849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:33705 (local address 127.0.0.1:59384)
0808 02:27:35.015319 (+ 18470us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:35.015349 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:35.015570 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:35.017959 (+ 2389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:35.017964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:35.017995 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:35.018724 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.018743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.022255 (+ 3512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.022261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.026706 (+ 4445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.026721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.028562 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.028568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.028760 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.028793 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 02:27:35.028883 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 02:27:35.029017 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18344,"thread_start_us":18150,"threads_started":1}
I20260808 02:27:35.029626 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:34.996856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.19.231.2:33705)
0808 02:27:35.011578 (+ 14722us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.011587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.015457 (+ 3870us) server_negotiation.cc:407] Connection header received
0808 02:27:35.017214 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.017231 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.017493 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.017665 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:35.018972 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.022039 (+ 3067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.026858 (+ 4819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.028310 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.028462 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.028702 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.029004 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.029188 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14535,"thread_start_us":14732,"threads_started":1}
I20260808 02:27:35.029948 1522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98cdedef5e79436fabc52385b4adb25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:35.029395 1817 tablet_service.cc:1467] Processing CreateTablet for tablet 98cdedef5e79436fabc52385b4adb25b (DEFAULT_TABLE table=test-table [id=1f6a225a0c9f4f3d8a730234b91ad9ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:35.032354 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98cdedef5e79436fabc52385b4adb25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:35.032968 1673 tablet_service.cc:1467] Processing CreateTablet for tablet 98cdedef5e79436fabc52385b4adb25b (DEFAULT_TABLE table=test-table [id=1f6a225a0c9f4f3d8a730234b91ad9ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:35.034327 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98cdedef5e79436fabc52385b4adb25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:35.042181 1919 tablet_bootstrap.cc:492] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Bootstrap starting.
I20260808 02:27:35.045521 1919 tablet_bootstrap.cc:654] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:35.046564 1919 log.cc:824] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:35.049605 1919 tablet_bootstrap.cc:492] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: No bootstrap required, opened a new log
I20260808 02:27:35.049726 1922 tablet_bootstrap.cc:492] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Bootstrap starting.
I20260808 02:27:35.049908 1919 ts_tablet_manager.cc:1397] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260808 02:27:35.049897 1921 tablet_bootstrap.cc:492] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Bootstrap starting.
I20260808 02:27:35.053272 1921 tablet_bootstrap.cc:654] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:35.053996 1922 tablet_bootstrap.cc:654] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:35.054471 1921 log.cc:824] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:35.055313 1922 log.cc:824] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:35.063269 1919 raft_consensus.cc:348] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.063871 1919 raft_consensus.cc:374] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:35.064049 1919 raft_consensus.cc:729] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bae0ad0b25d49c59f90f6af2e61d91a, State: Initialized, Role: FOLLOWER
I20260808 02:27:35.064740 1922 tablet_bootstrap.cc:492] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: No bootstrap required, opened a new log
I20260808 02:27:35.064883 1919 consensus_queue.cc:260] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.065080 1922 ts_tablet_manager.cc:1397] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260808 02:27:35.068182 1921 tablet_bootstrap.cc:492] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: No bootstrap required, opened a new log
I20260808 02:27:35.068478 1921 ts_tablet_manager.cc:1397] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20260808 02:27:35.075398 1593 heartbeater.cc:503] Master 127.19.231.62:43009 was elected leader, sending a full tablet report...
I20260808 02:27:35.076150 1919 ts_tablet_manager.cc:1428] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Time spent starting tablet: real 0.026s user 0.020s sys 0.003s
I20260808 02:27:35.077674 1921 raft_consensus.cc:348] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.078055 1921 raft_consensus.cc:374] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:35.078176 1921 raft_consensus.cc:729] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c468be546594993a9c0bd1e66d3f513, State: Initialized, Role: FOLLOWER
I20260808 02:27:35.078835 1921 consensus_queue.cc:260] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.078701 1922 raft_consensus.cc:348] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.079123 1922 raft_consensus.cc:374] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:35.079262 1922 raft_consensus.cc:729] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddcd7ae5001847c09d194513bbf43233, State: Initialized, Role: FOLLOWER
I20260808 02:27:35.079921 1922 consensus_queue.cc:260] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.080549 1888 heartbeater.cc:503] Master 127.19.231.62:43009 was elected leader, sending a full tablet report...
I20260808 02:27:35.081349 1921 ts_tablet_manager.cc:1428] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260808 02:27:35.083906 1922 ts_tablet_manager.cc:1428] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260808 02:27:35.089003 1926 raft_consensus.cc:482] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:27:35.089263 1926 raft_consensus.cc:504] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.090938 1926 leader_election.cc:288] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bae0ad0b25d49c59f90f6af2e61d91a (127.19.231.1:34453), ddcd7ae5001847c09d194513bbf43233 (127.19.231.2:33705)
I20260808 02:27:35.097973 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.091555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:40673 (local address 127.19.231.2:33705)
0808 02:27:35.091797 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.091804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.091822 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:27:35.091903 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.091908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.091980 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.092054 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:35.092520 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.093518 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.095895 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.097098 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.097264 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.097340 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.097612 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.097727 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 02:27:35.097987 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.091204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:33705 (local address 127.19.231.3:40673)
0808 02:27:35.091436 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:35.091457 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:35.091600 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:35.092103 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:35.092109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:35.092130 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:35.092355 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.092363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.093655 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.093658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.095750 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.095764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.097255 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.097259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.097378 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.097436 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 02:27:35.097566 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 02:27:35.097715 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260808 02:27:35.098249 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.091212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:36413 (local address 127.19.231.1:34453)
0808 02:27:35.091433 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.091440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.092106 (+ 666us) server_negotiation.cc:407] Connection header received
0808 02:27:35.092255 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.092269 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.092334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.092404 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:35.092899 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.093831 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.096101 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.097332 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.097472 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.097520 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.097857 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.097986 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:27:35.098276 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.091285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:34453 (local address 127.19.231.3:36413)
0808 02:27:35.091983 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:35.091996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:35.092126 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:35.092443 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:35.092450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:35.092478 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:35.092751 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.092759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.093959 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.093964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.095970 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:35.095980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.097534 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.097540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:35.097702 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.097729 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 02:27:35.097814 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 02:27:35.097926 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":413,"threads_started":1}
I20260808 02:27:35.099318 1694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98cdedef5e79436fabc52385b4adb25b" candidate_uuid: "6c468be546594993a9c0bd1e66d3f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddcd7ae5001847c09d194513bbf43233" is_pre_election: true
I20260808 02:27:35.099318 1544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98cdedef5e79436fabc52385b4adb25b" candidate_uuid: "6c468be546594993a9c0bd1e66d3f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" is_pre_election: true
I20260808 02:27:35.099920 1544 raft_consensus.cc:2454] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c468be546594993a9c0bd1e66d3f513 in term 0.
I20260808 02:27:35.099920 1694 raft_consensus.cc:2454] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c468be546594993a9c0bd1e66d3f513 in term 0.
I20260808 02:27:35.100647 1769 leader_election.cc:302] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c468be546594993a9c0bd1e66d3f513, ddcd7ae5001847c09d194513bbf43233; no voters:
I20260808 02:27:35.101101 1926 raft_consensus.cc:2781] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:27:35.101285 1926 raft_consensus.cc:482] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:27:35.101397 1926 raft_consensus.cc:3037] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:35.103715 1926 raft_consensus.cc:504] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.104287 1926 leader_election.cc:288] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [CANDIDATE]: Term 1 election: Requested vote from peers 7bae0ad0b25d49c59f90f6af2e61d91a (127.19.231.1:34453), ddcd7ae5001847c09d194513bbf43233 (127.19.231.2:33705)
I20260808 02:27:35.104727 1544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98cdedef5e79436fabc52385b4adb25b" candidate_uuid: "6c468be546594993a9c0bd1e66d3f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a"
I20260808 02:27:35.104779 1694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98cdedef5e79436fabc52385b4adb25b" candidate_uuid: "6c468be546594993a9c0bd1e66d3f513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddcd7ae5001847c09d194513bbf43233"
I20260808 02:27:35.104923 1544 raft_consensus.cc:3037] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:35.105020 1694 raft_consensus.cc:3037] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:35.107534 1694 raft_consensus.cc:2454] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c468be546594993a9c0bd1e66d3f513 in term 1.
I20260808 02:27:35.107576 1544 raft_consensus.cc:2454] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c468be546594993a9c0bd1e66d3f513 in term 1.
I20260808 02:27:35.107945 1769 leader_election.cc:302] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c468be546594993a9c0bd1e66d3f513, ddcd7ae5001847c09d194513bbf43233; no voters:
I20260808 02:27:35.108227 1926 raft_consensus.cc:2781] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:27:35.108680 1926 raft_consensus.cc:686] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 1 LEADER]: Becoming Leader. State: Replica: 6c468be546594993a9c0bd1e66d3f513, State: Running, Role: LEADER
I20260808 02:27:35.109143 1926 consensus_queue.cc:237] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } }
I20260808 02:27:35.114287 1387 catalog_manager.cc:5495] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c468be546594993a9c0bd1e66d3f513 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "6c468be546594993a9c0bd1e66d3f513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c468be546594993a9c0bd1e66d3f513" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 } health_report { overall_health: UNKNOWN } } }
W20260808 02:27:35.123143 1893 tablet.cc:2369] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:27:35.132970 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.127285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.19.231.3:44259)
0808 02:27:35.127512 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.127518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.127652 (+ 134us) server_negotiation.cc:407] Connection header received
0808 02:27:35.127894 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.127898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.127971 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.128164 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:27:35.128625 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.129354 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.130774 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.131228 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.131377 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.131900 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:27:35.132401 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:27:35.132569 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.132623 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.132702 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:27:35.141319 1544 raft_consensus.cc:1261] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 1 FOLLOWER]: Refusing update from remote peer 6c468be546594993a9c0bd1e66d3f513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:35.141319 1694 raft_consensus.cc:1261] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 1 FOLLOWER]: Refusing update from remote peer 6c468be546594993a9c0bd1e66d3f513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:35.142401 1929 consensus_queue.cc:1035] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddcd7ae5001847c09d194513bbf43233" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:27:35.142880 1926 consensus_queue.cc:1035] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bae0ad0b25d49c59f90f6af2e61d91a" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:27:35.154928 1934 mvcc.cc:205] Tried to move back new op lower bound from 7316095201848139776 to 7316095201731358720. Current Snapshot: MvccSnapshot[applied={T|T < 7316095201848139776}]
I20260808 02:27:35.170392 1387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:27:35.170758 1387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:27:35.175196 1387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1adc42c7c4b64981a25cbf1c79e861a0: Sending DeleteTablet for 3 replicas of tablet 98cdedef5e79436fabc52385b4adb25b
I20260808 02:27:35.176461 1817 tablet_service.cc:1514] Processing DeleteTablet for tablet 98cdedef5e79436fabc52385b4adb25b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:35 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:40518
I20260808 02:27:35.176456 1673 tablet_service.cc:1514] Processing DeleteTablet for tablet 98cdedef5e79436fabc52385b4adb25b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:35 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:59384
I20260808 02:27:35.176445 1522 tablet_service.cc:1514] Processing DeleteTablet for tablet 98cdedef5e79436fabc52385b4adb25b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:35 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:34990
I20260808 02:27:35.185779 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 1425
I20260808 02:27:35.191537 1944 tablet_replica.cc:330] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: stopping tablet replica
I20260808 02:27:35.192277 1944 raft_consensus.cc:2229] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:27:35.193039 1944 raft_consensus.cc:2258] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:35.194113 1943 tablet_replica.cc:330] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: stopping tablet replica
I20260808 02:27:35.194404 1942 tablet_replica.cc:330] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: stopping tablet replica
I20260808 02:27:35.194695 1943 raft_consensus.cc:2229] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:27:35.194871 1942 raft_consensus.cc:2229] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:27:35.195372 1942 raft_consensus.cc:2258] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:35.195520 1943 raft_consensus.cc:2258] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:35.197479 1944 ts_tablet_manager.cc:1905] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:35.197842 1943 ts_tablet_manager.cc:1905] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:35.200382 1942 ts_tablet_manager.cc:1905] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:35.206244 1943 ts_tablet_manager.cc:1918] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:35.206450 1943 log.cc:1196] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/98cdedef5e79436fabc52385b4adb25b
I20260808 02:27:35.207178 1943 ts_tablet_manager.cc:1939] T 98cdedef5e79436fabc52385b4adb25b P 6c468be546594993a9c0bd1e66d3f513: Deleting consensus metadata
I20260808 02:27:35.209381 1371 catalog_manager.cc:4841] TS 6c468be546594993a9c0bd1e66d3f513 (127.19.231.3:44259): tablet 98cdedef5e79436fabc52385b4adb25b (table test-table [id=1f6a225a0c9f4f3d8a730234b91ad9ac]) successfully deleted
I20260808 02:27:35.211335 1944 ts_tablet_manager.cc:1918] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:35.211510 1944 log.cc:1196] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/98cdedef5e79436fabc52385b4adb25b
I20260808 02:27:35.212165 1944 ts_tablet_manager.cc:1939] T 98cdedef5e79436fabc52385b4adb25b P 7bae0ad0b25d49c59f90f6af2e61d91a: Deleting consensus metadata
I20260808 02:27:35.212898 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.190319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.19.231.1:34453)
0808 02:27:35.190569 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.190579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.190627 (+ 48us) server_negotiation.cc:407] Connection header received
0808 02:27:35.190720 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.190728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.190841 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.190983 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:35.193256 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.194499 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.200097 (+ 5598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.200803 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.200968 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.204577 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:35.204593 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:35.204610 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:35.204658 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:35.208856 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:35.209817 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:35.209826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:35.209830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:35.209998 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:35.210834 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:35.210839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:35.210843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:35.211367 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:35.211794 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.212096 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.212368 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 02:27:35.213778 1588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:35.215763 1942 ts_tablet_manager.cc:1918] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:35.215991 1942 log.cc:1196] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/98cdedef5e79436fabc52385b4adb25b
I20260808 02:27:35.217510 1942 ts_tablet_manager.cc:1939] T 98cdedef5e79436fabc52385b4adb25b P ddcd7ae5001847c09d194513bbf43233: Deleting consensus metadata
I20260808 02:27:35.219594 1370 catalog_manager.cc:4841] TS ddcd7ae5001847c09d194513bbf43233 (127.19.231.2:33705): tablet 98cdedef5e79436fabc52385b4adb25b (table test-table [id=1f6a225a0c9f4f3d8a730234b91ad9ac]) successfully deleted
I20260808 02:27:35.310365 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 1425
I20260808 02:27:35.310858 1372 catalog_manager.cc:4841] TS 7bae0ad0b25d49c59f90f6af2e61d91a (127.19.231.1:34453): tablet 98cdedef5e79436fabc52385b4adb25b (table test-table [id=1f6a225a0c9f4f3d8a730234b91ad9ac]) successfully deleted
I20260808 02:27:35.338788 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 1598
I20260808 02:27:35.357255 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.339695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.19.231.2:33705)
0808 02:27:35.340345 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.340360 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.340379 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:27:35.340449 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.340455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.340535 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.340672 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:35.342459 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.343686 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.345574 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.346234 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.346375 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.349770 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:35.349791 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:35.349811 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:35.349861 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:35.354309 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:35.355032 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:35.355042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:35.355046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:35.355232 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:35.355639 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:35.355644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:35.355648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:35.356123 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:35.356437 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.356619 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.356853 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498}
I20260808 02:27:35.358000 1739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:35.451583 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 1598
I20260808 02:27:35.479599 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 1748
I20260808 02:27:35.494831 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:35.480526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.19.231.3:44259)
0808 02:27:35.480784 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:35.480792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:35.480812 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:27:35.480887 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:35.480890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:35.480959 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:35.481080 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:35.482478 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.483464 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.485014 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:35.485524 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:35.485666 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:35.488543 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:35.488554 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:35.488570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:35.488615 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:35.491745 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:35.492523 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:35.492531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:35.492533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:35.492704 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:35.493119 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:35.493126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:35.493131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:35.493508 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:35.493906 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:35.494194 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 02:27:35.494421 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260808 02:27:35.495675 1883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:35.620321 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 1748
I20260808 02:27:35.641443 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 1352
I20260808 02:27:35.642778 1418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:35.757622 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 1352
I20260808 02:27:35.786923 20380 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 02:27:36.132378 20380 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 02:27:36.132668 20380 mini_ranger.cc:66] Stopping Ranger...
I20260808 02:27:36.487996 20380 mini_ranger.cc:68] Stopped Ranger
2026-08-08 02:27:36.488 UTC [365] LOG: received smart shutdown request
2026-08-08 02:27:36.491 UTC [365] LOG: background worker "logical replication launcher" (PID 374) exited with exit code 1
2026-08-08 02:27:36.491 UTC [369] LOG: shutting down
2026-08-08 02:27:36.664 UTC [365] LOG: database system is shut down
2026-08-08T02:27:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55839 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:27:36 dist-test-slave-pgkr krb5kdc[1957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:27:36 dist-test-slave-pgkr krb5kdc[1957](info): set up 2 sockets
Aug 08 02:27:36 dist-test-slave-pgkr krb5kdc[1957](info): commencing operation
krb5kdc: starting...
W20260808 02:27:38.010353 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:27:38 dist-test-slave-pgkr krb5kdc[1957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:27:38.298815 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44179
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:44179,127.19.231.61:42055,127.19.231.60:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:38.650998 1971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:38.651292 1971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:38.651484 1971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:38.658875 1971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:38.658980 1971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:38.659085 1971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:38.659194 1971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:38.659279 1971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:38.669299 1971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:44179,127.19.231.61:42055,127.19.231.60:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:38.670724 1971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:38.672091 1971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:38.684389 1977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:38.684518 1978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:38.684478 1980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:38.685675 1971 server_base.cc:1034] running on GCE node
I20260808 02:27:38.686386 1971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:38.687795 1971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:38.689005 1971 hybrid_clock.cc:648] HybridClock initialized: now 1786156058688965 us; error 35 us; skew 500 ppm
I20260808 02:27:38.689472 1971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:38 dist-test-slave-pgkr krb5kdc[1957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:38.693009 1971 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:27:38.694566 1971 webserver.cc:460] Webserver started at http://127.19.231.62:45839/ using document root <none> and password file <none>
I20260808 02:27:38.695289 1971 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:38.695444 1971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:38.695770 1971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:38.698828 1971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "1b2a02224cb242cba1e2eb51a355e4d1"
format_stamp: "Formatted at 2026-08-08 02:27:38 on dist-test-slave-pgkr"
server_key: "0b2c9f1ec60148915310ffcba4f48341"
server_key_iv: "0b164f93619be4f837bf9712f7c245c7"
server_key_version: "encryptionkey@0"
I20260808 02:27:38.699671 1971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "1b2a02224cb242cba1e2eb51a355e4d1"
format_stamp: "Formatted at 2026-08-08 02:27:38 on dist-test-slave-pgkr"
server_key: "0b2c9f1ec60148915310ffcba4f48341"
server_key_iv: "0b164f93619be4f837bf9712f7c245c7"
server_key_version: "encryptionkey@0"
I20260808 02:27:38.705338 1971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:27:38.709003 1987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:38.710445 1971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:27:38.710704 1971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "1b2a02224cb242cba1e2eb51a355e4d1"
format_stamp: "Formatted at 2026-08-08 02:27:38 on dist-test-slave-pgkr"
server_key: "0b2c9f1ec60148915310ffcba4f48341"
server_key_iv: "0b164f93619be4f837bf9712f7c245c7"
server_key_version: "encryptionkey@0"
I20260808 02:27:38.710906 1971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:38.751263 1971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:38.756106 1971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:38.756353 1971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:38.776787 1971 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:44179
I20260808 02:27:38.776818 2040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:44179 every 8 connection(s)
I20260808 02:27:38.778214 1971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:27:38.779057 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 1971
I20260808 02:27:38.779327 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:27:38.780092 20380 external_mini_cluster.cc:1428] Setting key 2106b534ec2b62bb793ad5e18edea96b
I20260808 02:27:38.782332 2041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:38.789000 2041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.231.61@KRBTEST.COM; defaulting to no policy
Aug 08 02:27:38 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.61@KRBTEST.COM" created.
I20260808 02:27:38.819178 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.793210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.19.231.62:44179)
0808 02:27:38.793989 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:38.794000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:38.794059 (+ 59us) server_negotiation.cc:407] Connection header received
0808 02:27:38.795237 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:38.795246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:38.795544 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:38.795772 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:38.796931 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:38.799108 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:38.800701 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:38.801690 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:38.801884 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:38.810850 (+ 8966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:38.810860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:38.810873 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:38.810907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:38.814178 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:38.815300 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:38.815307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:38.815309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:38.815459 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:38.815977 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:38.815980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:38.815982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:38.816537 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:38.816707 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:38.817479 (+ 772us) server_negotiation.cc:299] Negotiation successful
0808 02:27:38.817813 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":552,"thread_start_us":366,"threads_started":1}
I20260808 02:27:38.819278 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.792143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:44179 (local address 127.0.0.1:33124)
0808 02:27:38.792883 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:38.792935 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:38.794873 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:38.795801 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:38.795806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:38.796368 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:38.796723 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:38.796742 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:38.799297 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:38.799301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:38.800539 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:38.800553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:38.801941 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:38.801946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:38.802183 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:38.807953 (+ 5770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:38.807979 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:38.810411 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:38.814339 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:38.814358 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:38.814375 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:38.815165 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:38.815591 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:38.815599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:38.815605 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:38.815852 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:38.816698 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:38.816711 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:38.817009 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 02:27:38.817359 (+ 350us) client_negotiation.cc:240] Negotiation successful
0808 02:27:38.818035 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":94,"thread_start_us":296,"threads_started":1}
I20260808 02:27:38.822387 2041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:27:38.823601 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.823049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42104)
0808 02:27:38.823270 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:38.823414 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 02:27:38.823894 1989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
I20260808 02:27:38.825723 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.825174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42116)
0808 02:27:38.825378 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:38.825481 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 02:27:38.825935 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:42055: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab.
Entry for principal kudu/127.19.231.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab.
I20260808 02:27:38.850766 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:42055
--webserver_interface=127.19.231.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab
--principal=kudu/127.19.231.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:44179,127.19.231.61:42055,127.19.231.60:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:27:38.875310 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } attempt: 1
I20260808 02:27:38.876618 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.876135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42124)
0808 02:27:38.876356 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:38.876477 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 02:27:38.877744 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.877320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42126)
0808 02:27:38.877474 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:38.877598 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:27:38.878206 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:42055: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
I20260808 02:27:38.946513 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } attempt: 2
I20260808 02:27:38.947813 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.947313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42142)
0808 02:27:38.947575 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:38.947671 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 02:27:38.948979 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:38.948596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42144)
0808 02:27:38.948766 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:38.948850 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:27:38.949237 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:42055: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
I20260808 02:27:39.040525 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } attempt: 3
I20260808 02:27:39.041872 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.041309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42146)
0808 02:27:39.041536 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.041727 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 02:27:39.043169 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.042785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42150)
0808 02:27:39.042945 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.043016 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:27:39.043375 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:42055: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
I20260808 02:27:39.186679 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } attempt: 4
I20260808 02:27:39.187969 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.187377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42162)
0808 02:27:39.187744 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.187834 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260808 02:27:39.189098 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.188753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42178)
0808 02:27:39.188895 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.188983 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 02:27:39.189388 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:42055: Network error: Client connection negotiation failed: client connection to 127.19.231.61:42055: connect: Connection refused (error 111)
W20260808 02:27:39.217530 2048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:39.217813 2048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:39.217991 2048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:39.225554 2048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:39.225663 2048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:39.225764 2048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:39.225852 2048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:39.225919 2048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:39.236038 2048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:44179,127.19.231.61:42055,127.19.231.60:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab
--principal=kudu/127.19.231.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.231.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:39.237442 2048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:39.238862 2048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:39.251962 2054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:39.253068 2048 server_base.cc:1034] running on GCE node
W20260808 02:27:39.253269 2053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:39.257346 2056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:39.258479 2048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:39.260142 2048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:39.261524 2048 hybrid_clock.cc:648] HybridClock initialized: now 1786156059261474 us; error 38 us; skew 500 ppm
I20260808 02:27:39.261947 2048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:39.265233 2048 init.cc:377] Logged in from keytab as kudu/127.19.231.61@KRBTEST.COM (short username kudu)
I20260808 02:27:39.266752 2048 webserver.cc:460] Webserver started at http://127.19.231.61:40199/ using document root <none> and password file <none>
I20260808 02:27:39.267462 2048 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:39.267597 2048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:39.267943 2048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:39.271222 2048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data/instance:
uuid: "e6d9a04205634b2d9afbbed40651c273"
format_stamp: "Formatted at 2026-08-08 02:27:39 on dist-test-slave-pgkr"
server_key: "63b93b963640e656d839f7da49123084"
server_key_iv: "f04a11125b57aab16573e3148c4d95b7"
server_key_version: "encryptionkey@0"
I20260808 02:27:39.272071 2048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/wal/instance:
uuid: "e6d9a04205634b2d9afbbed40651c273"
format_stamp: "Formatted at 2026-08-08 02:27:39 on dist-test-slave-pgkr"
server_key: "63b93b963640e656d839f7da49123084"
server_key_iv: "f04a11125b57aab16573e3148c4d95b7"
server_key_version: "encryptionkey@0"
I20260808 02:27:39.277474 2048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:27:39.281309 2064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:39.282828 2048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:27:39.283005 2048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/wal
uuid: "e6d9a04205634b2d9afbbed40651c273"
format_stamp: "Formatted at 2026-08-08 02:27:39 on dist-test-slave-pgkr"
server_key: "63b93b963640e656d839f7da49123084"
server_key_iv: "f04a11125b57aab16573e3148c4d95b7"
server_key_version: "encryptionkey@0"
I20260808 02:27:39.283200 2048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:39.311306 2048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:39.315742 2048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:39.316043 2048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:39.336333 2048 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.61:42055
I20260808 02:27:39.336370 2120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.61:42055 every 8 connection(s)
I20260808 02:27:39.337760 2048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
I20260808 02:27:39.341300 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2048
I20260808 02:27:39.341512 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-1/wal/instance
I20260808 02:27:39.341825 2121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:39.342042 20380 external_mini_cluster.cc:1428] Setting key 499311bc1c6acc7cf213ddf063381aae
I20260808 02:27:39.348651 2121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.231.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.60@KRBTEST.COM" created.
I20260808 02:27:39.375439 2123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.351421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.19.231.62:44179)
0808 02:27:39.352034 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.352039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.353124 (+ 1085us) server_negotiation.cc:407] Connection header received
0808 02:27:39.356243 (+ 3119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.356249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.356361 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.356474 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.358801 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.359769 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.362480 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.362991 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.363099 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.367728 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.367736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.367741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.367795 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.371855 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.372782 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.372786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.372788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.372913 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.373378 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.373387 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.373390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.373760 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.373883 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.374807 (+ 924us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.374984 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":385,"threads_started":1}
I20260808 02:27:39.376421 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.352063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:44179 (local address 127.0.0.1:33134)
0808 02:27:39.352911 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.352952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:39.355911 (+ 2959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:39.356757 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:39.356770 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:39.357667 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:39.358521 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.358564 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.359958 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.359973 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.362227 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.362247 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.363229 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.363239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.363648 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.364437 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:39.364488 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:39.367428 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:39.372023 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.372030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.372044 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.372619 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.373009 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.373013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.373015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.373220 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.373882 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:39.373889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:39.374244 (+ 355us) client_negotiation.cc:769] Sending connection context
0808 02:27:39.374649 (+ 405us) client_negotiation.cc:240] Negotiation successful
0808 02:27:39.375155 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":392,"threads_started":1}
I20260808 02:27:39.378324 2121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:27:39.394479 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.379088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42180)
0808 02:27:39.379255 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.379277 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:39.379375 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:39.380529 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:39.380533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:39.380992 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:39.381123 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.381131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.382498 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.382502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.383421 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.383432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.384240 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.384246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.384476 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.385060 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:39.385077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:39.387542 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:39.392127 (+ 4585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.392131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.392133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.392511 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.392918 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.392921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.392923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.393071 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.393798 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:39.393801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:39.393912 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 02:27:39.394018 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 02:27:39.394210 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":3}
I20260808 02:27:39.395251 2126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.379424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.19.231.61:42055)
0808 02:27:39.380121 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.380127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.380152 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:27:39.380202 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.380208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.380387 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.380547 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.381291 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.382365 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.383555 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.384089 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.384235 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.387757 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.387772 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.387790 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.387832 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.391978 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.392643 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.392654 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.392659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.392807 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.393184 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.393191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.393197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.393677 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.393899 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.394251 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.394552 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":410,"threads_started":1}
I20260808 02:27:39.397838 2121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:27:39.399082 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.398664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41864)
0808 02:27:39.398840 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.398941 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:27:39.399354 2068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.401077 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.400761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41872)
0808 02:27:39.400894 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.400971 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:27:39.401491 2121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab.
Entry for principal kudu/127.19.231.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab.
I20260808 02:27:39.411813 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:38615
--webserver_interface=127.19.231.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab
--principal=kudu/127.19.231.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:44179,127.19.231.61:42055,127.19.231.60:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:27:39.450975 2121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 1
I20260808 02:27:39.452316 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.451786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41882)
0808 02:27:39.451997 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.452122 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 02:27:39.453454 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.453053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41892)
0808 02:27:39.453239 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.453309 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:27:39.453737 2121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.488688 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } attempt: 5
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:27:39.503351 2126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.489514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.19.231.61:42055)
0808 02:27:39.489707 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.489713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.489739 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:27:39.489896 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.489901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.490005 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.490122 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.491016 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.492109 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.493308 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.494089 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.494231 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.497849 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.497855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.497858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.497890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.500685 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.501492 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.501496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.501499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.501626 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.502056 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.502059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.502061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.502346 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.502461 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.502928 (+ 467us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.503131 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:27:39.503611 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.489431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42192)
0808 02:27:39.489631 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.489653 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:39.489767 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:39.490184 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:39.490188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:39.490665 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:39.490849 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.490858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.492289 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.492293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.493162 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.493175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.494261 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.494266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.494406 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.495073 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:39.495092 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:39.497627 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:39.500826 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.500832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.500836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.501363 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.501738 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.501743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.501747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.501921 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.502469 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:39.502473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:39.502663 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 02:27:39.502839 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 02:27:39.503157 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:27:39.504702 2041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:27:39.505913 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.505515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41906)
0808 02:27:39.505705 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.505778 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 02:27:39.506922 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.506600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41916)
0808 02:27:39.506751 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.506833 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 02:27:39.507158 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.521996 2121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 2
I20260808 02:27:39.523203 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.522750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41922)
0808 02:27:39.522942 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.523044 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:27:39.524219 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.523889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41936)
0808 02:27:39.524036 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.524105 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:27:39.524449 2121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.558401 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 1
I20260808 02:27:39.559752 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.559243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41948)
0808 02:27:39.559441 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.559585 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 02:27:39.560788 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.560442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41964)
0808 02:27:39.560610 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.560690 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 02:27:39.561034 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.615713 2121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 3
I20260808 02:27:39.617013 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.616520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41978)
0808 02:27:39.616738 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.616842 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:27:39.618261 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.617785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41982)
0808 02:27:39.618052 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.618127 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260808 02:27:39.618526 2121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.629277 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 2
I20260808 02:27:39.630517 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.630002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41996)
0808 02:27:39.630236 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.630342 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 02:27:39.631580 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.631240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:41998)
0808 02:27:39.631388 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.631474 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 02:27:39.631906 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.738195 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 3
I20260808 02:27:39.739580 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.738961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:42012)
0808 02:27:39.739216 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.739342 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:27:39.740866 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.740307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:42018)
0808 02:27:39.740566 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.740708 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260808 02:27:39.741139 2041 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
I20260808 02:27:39.761871 2121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 4
I20260808 02:27:39.763401 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.762798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:42026)
0808 02:27:39.763122 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.763231 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 02:27:39.764600 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.764194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:42038)
0808 02:27:39.764369 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.764446 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 02:27:39.764828 2121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:38615: Network error: Client connection negotiation failed: client connection to 127.19.231.60:38615: connect: Connection refused (error 111)
W20260808 02:27:39.770825 2130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:39.771142 2130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:39.771245 2130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:39.778923 2130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:39.779037 2130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:39.779196 2130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:39.779294 2130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:39.779378 2130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:39.790025 2130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:44179,127.19.231.61:42055,127.19.231.60:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab
--principal=kudu/127.19.231.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.231.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:39.791568 2130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:39.792953 2130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:39.805599 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:39.805559 2136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:39.805684 2135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:39.811302 2130 server_base.cc:1034] running on GCE node
I20260808 02:27:39.812247 2130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:39.813710 2130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:39.814924 2130 hybrid_clock.cc:648] HybridClock initialized: now 1786156059814872 us; error 36 us; skew 500 ppm
I20260808 02:27:39.815356 2130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:39.819105 2130 init.cc:377] Logged in from keytab as kudu/127.19.231.60@KRBTEST.COM (short username kudu)
I20260808 02:27:39.820887 2130 webserver.cc:460] Webserver started at http://127.19.231.60:38565/ using document root <none> and password file <none>
I20260808 02:27:39.821611 2130 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:39.821722 2130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:39.821976 2130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:39.825064 2130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data/instance:
uuid: "ab0344fe51724f08b8ec8011f569ae1e"
format_stamp: "Formatted at 2026-08-08 02:27:39 on dist-test-slave-pgkr"
server_key: "9e07d15dd3ccfa47f470de4d22dcae4c"
server_key_iv: "069e52226828742c2a59b355d2b75a47"
server_key_version: "encryptionkey@0"
I20260808 02:27:39.825865 2130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/wal/instance:
uuid: "ab0344fe51724f08b8ec8011f569ae1e"
format_stamp: "Formatted at 2026-08-08 02:27:39 on dist-test-slave-pgkr"
server_key: "9e07d15dd3ccfa47f470de4d22dcae4c"
server_key_iv: "069e52226828742c2a59b355d2b75a47"
server_key_version: "encryptionkey@0"
I20260808 02:27:39.831439 2130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:27:39.835018 2145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:39.836370 2130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:27:39.836516 2130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/wal
uuid: "ab0344fe51724f08b8ec8011f569ae1e"
format_stamp: "Formatted at 2026-08-08 02:27:39 on dist-test-slave-pgkr"
server_key: "9e07d15dd3ccfa47f470de4d22dcae4c"
server_key_iv: "069e52226828742c2a59b355d2b75a47"
server_key_version: "encryptionkey@0"
I20260808 02:27:39.836695 2130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:39.855425 2130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:39.860139 2130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:39.860399 2130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:39.880573 2130 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.60:38615
I20260808 02:27:39.880607 2199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.60:38615 every 8 connection(s)
I20260808 02:27:39.882058 2130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
I20260808 02:27:39.885967 2200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:39.891503 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2130
I20260808 02:27:39.891695 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786155839876607-20380-0/minicluster-data/master-2/wal/instance
I20260808 02:27:39.892295 20380 external_mini_cluster.cc:1428] Setting key b42dfb77f9e6d06dde5af46708f68466
I20260808 02:27:39.893319 2200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:27:39.913800 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.896299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.19.231.62:44179)
0808 02:27:39.896912 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.896918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.899058 (+ 2140us) server_negotiation.cc:407] Connection header received
0808 02:27:39.899267 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.899273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.899346 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.899463 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.901037 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.901862 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.903420 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.903831 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.903928 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.907489 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.907494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.907498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.907530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.910507 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.911365 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.911371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.911374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.911536 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.911981 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.911990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.911992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.912277 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.912437 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.913171 (+ 734us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.913393 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":297,"threads_started":1}
I20260808 02:27:39.914988 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.894293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.19.231.60:38615)
0808 02:27:39.895206 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.895218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.895261 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:27:39.897378 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.897400 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.897790 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.898088 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.899983 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.901770 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.903326 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.903923 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.904120 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.908177 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.908185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.908197 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.908230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.911100 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.911862 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.911868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.911870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.912013 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.912450 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.912454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.912456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.912850 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.913170 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.913542 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.913816 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":495,"threads_started":1}
I20260808 02:27:39.915105 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.896368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:44179 (local address 127.0.0.1:33142)
0808 02:27:39.898922 (+ 2554us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.898958 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:39.899140 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:39.899532 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:39.899538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:39.900358 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:39.900848 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.900869 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.902016 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.902019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.903274 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.903292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.904063 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.904071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.904217 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.905054 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:39.905078 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:39.907299 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:39.910665 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.910679 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.910691 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.911232 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.911650 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.911655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.911659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.911854 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.912417 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:39.912425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:39.912777 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 02:27:39.913046 (+ 269us) client_negotiation.cc:240] Negotiation successful
0808 02:27:39.913563 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2363,"thread_start_us":2194,"threads_started":1}
I20260808 02:27:39.916747 2200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:27:39.918396 2041 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 4
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:39.933686 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.917413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:42055 (local address 127.0.0.1:42202)
0808 02:27:39.917609 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.917628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:39.917724 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:39.918150 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:39.918154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:39.918784 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:39.919020 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.919034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.920626 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.920633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.922230 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.922245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.923432 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.923436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.923622 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.924295 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:39.924320 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:39.926751 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:39.930347 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.930355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.930361 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.931016 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.931769 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.931775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.931781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.931962 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.932616 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:39.932622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:39.932795 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 02:27:39.932943 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 02:27:39.933260 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":28}
I20260808 02:27:39.933769 2126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.917590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.19.231.61:42055)
0808 02:27:39.917768 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.917773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.917805 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:27:39.917854 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.917858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.917964 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.918077 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.919265 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.920422 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.922391 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.923082 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.923212 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.926933 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.926938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.926941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.926974 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.930198 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.931430 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.931436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.931439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.931611 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.932120 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.932127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.932131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.932484 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.932670 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.933100 (+ 430us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.933364 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:27:39.934547 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.918916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:42056)
0808 02:27:39.919281 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.919299 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:39.919400 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:39.920086 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:39.920101 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:39.920591 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:39.920750 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.920759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.921806 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.921812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.923184 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.923197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.924217 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.924225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.924422 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.924991 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:39.925007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:39.928164 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:39.931687 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.931692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.931696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.932215 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.932666 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.932671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.932674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.932867 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.933578 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:39.933583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:39.933751 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 02:27:39.933884 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 02:27:39.934153 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260808 02:27:39.935025 2200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:27:39.935408 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.919625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.19.231.60:38615)
0808 02:27:39.919797 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.919803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.919818 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:27:39.919860 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.919865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.919930 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.920033 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.920886 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.921673 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.923330 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.924017 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.924724 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.928394 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.928401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.928405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.928445 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.931556 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.932377 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.932383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.932387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.932533 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.933015 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.933020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.933023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.933414 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.934759 (+ 1345us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.934885 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.935106 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":59}
I20260808 02:27:39.938719 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.918747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.19.231.62:44179)
0808 02:27:39.918934 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.918940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.919036 (+ 96us) server_negotiation.cc:407] Connection header received
0808 02:27:39.919391 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.919395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.919485 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.919692 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.921569 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.922589 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.924305 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.924950 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.925086 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.930579 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.930584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.930587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.930664 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.935619 (+ 4955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.936515 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.936522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.936526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.936682 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.937165 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.937172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.937177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.937466 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.937790 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.938081 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.938341 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 02:27:39.942291 2041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: Bootstrap starting.
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:27:39.946532 2041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:39.948026 2041 log.cc:824] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:39.950872 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.936180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.19.231.60:38615)
0808 02:27:39.936377 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.936382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.936839 (+ 457us) server_negotiation.cc:407] Connection header received
0808 02:27:39.937014 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.937019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.937100 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.937194 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.938213 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.939347 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.940952 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.941673 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.941794 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.945312 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.945316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.945320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.945347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.947898 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.948597 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.948605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.948608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.948767 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.949299 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.949303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.949306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.949670 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.949847 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.950201 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.950425 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:27:39.950939 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.936192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:42068)
0808 02:27:39.936767 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:39.936788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:39.936895 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:39.937383 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:39.937388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:39.937867 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:39.938050 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.938064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.939497 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.939503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.940772 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:39.940789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.942020 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.942028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:39.942226 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.942950 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:39.942976 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:39.945128 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:39.948027 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.948041 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.948045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.948481 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.948946 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:39.948953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:39.948957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:39.949155 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:39.949790 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:39.949794 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:39.949951 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 02:27:39.950117 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 02:27:39.950559 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":139}
I20260808 02:27:39.951668 2041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: No bootstrap required, opened a new log
Aug 08 02:27:39 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:27:39.955626 2200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e: Bootstrap starting.
I20260808 02:27:39.959146 2200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:39.960494 2126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.940743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.19.231.61:42055)
0808 02:27:39.940958 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.940964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.941045 (+ 81us) server_negotiation.cc:407] Connection header received
0808 02:27:39.941272 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.941279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.941356 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.941421 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.943461 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.944545 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.947042 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.947735 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.947841 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.953668 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.953675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.953680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.953722 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.957366 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.958227 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.958233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.958236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.958404 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:39.958893 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:39.958899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:39.958902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:39.959255 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:39.959546 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:39.959882 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 02:27:39.960135 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:27:39.960520 2200 log.cc:824] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:39.962919 2041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:39.963339 2041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:39.963497 2041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2a02224cb242cba1e2eb51a355e4d1, State: Initialized, Role: FOLLOWER
I20260808 02:27:39.963940 2200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e: No bootstrap required, opened a new log
I20260808 02:27:39.964264 2041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:39.965466 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:27:39.971465 2208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:39.972179 2208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:27:39.972839 2041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:39.977149 2200 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:39.977758 2200 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:39.977906 2200 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0344fe51724f08b8ec8011f569ae1e, State: Initialized, Role: FOLLOWER
I20260808 02:27:39.978853 2200 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:39.985957 2223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:39.986820 2223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:27:39.987514 2200 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:40.005935 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.977170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.19.231.60:38615)
0808 02:27:39.977343 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.977351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.977373 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:27:39.977648 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.977657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.977743 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.977821 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.980905 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.982416 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.985690 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.986407 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.986509 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:39.993137 (+ 6628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:39.993147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:39.993155 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:39.993208 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:39.997935 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.002698 (+ 4763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.002707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.002714 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.002913 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.003495 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.003503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.003509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.004016 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:40.004366 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:40.005094 (+ 728us) server_negotiation.cc:299] Negotiation successful
0808 02:27:40.005388 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 02:27:40.009479 1994 tablet.cc:2369] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:27:40.011494 2126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.980182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.19.231.61:42055)
0808 02:27:39.980360 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.980366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.987608 (+ 7242us) server_negotiation.cc:407] Connection header received
0808 02:27:39.988037 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.988046 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.988159 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.988274 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.990697 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.992134 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.995016 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.995930 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:39.996048 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:40.000320 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:40.000330 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:40.000338 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:40.000391 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:40.005063 (+ 4672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.006158 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.006163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.006166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.006322 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.009030 (+ 2708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.009037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.009040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.009529 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:40.009842 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:40.010594 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 02:27:40.010956 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:27:40.023712 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:39.976938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.19.231.62:44179)
0808 02:27:39.977125 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:39.977133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:39.979828 (+ 2695us) server_negotiation.cc:407] Connection header received
0808 02:27:39.981015 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:39.981023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:39.981126 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:39.981212 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:39.990021 (+ 8809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:39.991610 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:40.006995 (+ 15385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:40.007987 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:40.008130 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:40.014041 (+ 5911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:40.014067 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:40.014075 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:40.014161 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:40.019060 (+ 4899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.020215 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.020224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.020230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.020625 (+ 395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.021195 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.021204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.021208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.021687 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:40.022051 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:40.022832 (+ 781us) server_negotiation.cc:299] Negotiation successful
0808 02:27:40.023110 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:27:40.025838 2240 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260808 02:27:40.032503 2238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:27:40.032647 2238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:27:40.029812 1971 master_runner.cc:186] Error getting master registration for 127.19.231.61:42055: OK, instance_id { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" instance_seqno: 1786156059330538 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:27:40.032997 2130 master_runner.cc:186] Error getting master registration for 127.19.231.61:42055: OK, instance_id { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" instance_seqno: 1786156059330538 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:27:40.043469 2130 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:27:40.043829 1971 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:27:40.064071 2121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } attempt: 5
Aug 08 02:27:40 dist-test-slave-pgkr krb5kdc[1957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156059, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:27:40.080622 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:40.064857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.19.231.60:38615)
0808 02:27:40.065050 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:40.065056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:40.065078 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:27:40.065235 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:40.065238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:40.065302 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:40.065390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:40.066401 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:40.067597 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:40.069378 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:40.069799 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:40.069898 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:40.074138 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:40.074146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:40.074150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:40.074189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:40.077516 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.078389 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.078393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.078396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.078597 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:40.079059 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:40.079064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:40.079066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:40.079410 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:40.079598 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:40.080070 (+ 472us) server_negotiation.cc:299] Negotiation successful
0808 02:27:40.080277 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:27:40.080643 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:40.064755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:38615 (local address 127.0.0.1:42090)
0808 02:27:40.064978 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:40.065000 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:40.065112 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:40.065451 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:40.065456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:40.066024 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:40.066242 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:40.066255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:40.067733 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:40.067739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:40.069217 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:40.069233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:40.069950 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:40.069957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:40.070187 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:40.070887 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:40.070913 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:40.073910 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:40.077654 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:40.077660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:40.077665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:40.078242 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:40.078726 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:40.078732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:40.078736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:40.078907 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:40.079562 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:40.079568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:40.079746 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 02:27:40.079922 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 02:27:40.080258 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:27:40.086153 2121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: Bootstrap starting.
I20260808 02:27:40.090317 2121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:40.091652 2121 log.cc:824] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:40.094748 2121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: No bootstrap required, opened a new log
I20260808 02:27:40.103819 2121 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:40.104257 2121 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:40.104415 2121 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d9a04205634b2d9afbbed40651c273, State: Initialized, Role: FOLLOWER
I20260808 02:27:40.105053 2121 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:40.106114 2243 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.106539 2243 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:27:40.107448 2121 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 02:27:40.112661 2151 tablet.cc:2369] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:27:40.120329 2255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:27:40.120656 2255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:27:40.123148 2048 master_runner.cc:263] No leader master found from master e6d9a04205634b2d9afbbed40651c273
I20260808 02:27:40.123342 2048 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:27:40.178443 2223 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:27:40.178835 2223 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:40.180600 2223 leader_election.cc:288] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b2a02224cb242cba1e2eb51a355e4d1 (127.19.231.62:44179), e6d9a04205634b2d9afbbed40651c273 (127.19.231.61:42055)
I20260808 02:27:40.181604 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab0344fe51724f08b8ec8011f569ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d9a04205634b2d9afbbed40651c273" is_pre_election: true
I20260808 02:27:40.181739 2015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab0344fe51724f08b8ec8011f569ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" is_pre_election: true
I20260808 02:27:40.182250 2093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab0344fe51724f08b8ec8011f569ae1e in term 0.
I20260808 02:27:40.182458 2015 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab0344fe51724f08b8ec8011f569ae1e in term 0.
I20260808 02:27:40.183223 2147 leader_election.cc:302] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b2a02224cb242cba1e2eb51a355e4d1, ab0344fe51724f08b8ec8011f569ae1e; no voters:
I20260808 02:27:40.185079 2223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:27:40.185228 2223 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:27:40.185328 2223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:40.188766 2223 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:40.189451 2223 leader_election.cc:288] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [CANDIDATE]: Term 1 election: Requested vote from peers 1b2a02224cb242cba1e2eb51a355e4d1 (127.19.231.62:44179), e6d9a04205634b2d9afbbed40651c273 (127.19.231.61:42055)
I20260808 02:27:40.189918 2015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab0344fe51724f08b8ec8011f569ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b2a02224cb242cba1e2eb51a355e4d1"
I20260808 02:27:40.190162 2015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:40.193727 2015 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab0344fe51724f08b8ec8011f569ae1e in term 1.
I20260808 02:27:40.194206 2147 leader_election.cc:302] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b2a02224cb242cba1e2eb51a355e4d1, ab0344fe51724f08b8ec8011f569ae1e; no voters:
I20260808 02:27:40.194185 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab0344fe51724f08b8ec8011f569ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d9a04205634b2d9afbbed40651c273"
I20260808 02:27:40.194483 2093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:40.198382 2093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab0344fe51724f08b8ec8011f569ae1e in term 1.
I20260808 02:27:40.199095 2223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:27:40.199347 2223 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [term 1 LEADER]: Becoming Leader. State: Replica: ab0344fe51724f08b8ec8011f569ae1e, State: Running, Role: LEADER
I20260808 02:27:40.200083 2223 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } }
I20260808 02:27:40.209707 2257 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab0344fe51724f08b8ec8011f569ae1e. Latest consensus state: current_term: 1 leader_uuid: "ab0344fe51724f08b8ec8011f569ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.210119 2257 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:40.211022 2259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:27:40.217133 2259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:27:40.234323 2015 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [term 1 FOLLOWER]: Refusing update from remote peer ab0344fe51724f08b8ec8011f569ae1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:40.235430 2093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [term 1 FOLLOWER]: Refusing update from remote peer ab0344fe51724f08b8ec8011f569ae1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:40.236651 2257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:27:40.237319 2257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:27:40.275578 2208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab0344fe51724f08b8ec8011f569ae1e. Latest consensus state: current_term: 1 leader_uuid: "ab0344fe51724f08b8ec8011f569ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.275961 2208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:27:40.281559 2261 mvcc.cc:205] Tried to move back new op lower bound from 7316095222702616576 to 7316095222586544128. Current Snapshot: MvccSnapshot[applied={T|T < 7316095222702616576}]
I20260808 02:27:40.282589 2243 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab0344fe51724f08b8ec8011f569ae1e. Latest consensus state: current_term: 1 leader_uuid: "ab0344fe51724f08b8ec8011f569ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.284680 2243 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:27:40.288411 2243 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab0344fe51724f08b8ec8011f569ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.288682 2243 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:27:40.295828 2257 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab0344fe51724f08b8ec8011f569ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.296129 2257 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:40.296465 2257 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab0344fe51724f08b8ec8011f569ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.296797 2257 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:40.305402 2259 catalog_manager.cc:1347] Generated new cluster ID: 40da4c3cb2db46a091d9017091cf1642
I20260808 02:27:40.305537 2259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:27:40.306674 2208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab0344fe51724f08b8ec8011f569ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b2a02224cb242cba1e2eb51a355e4d1" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44179 } } peers { permanent_uuid: "e6d9a04205634b2d9afbbed40651c273" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 42055 } } peers { permanent_uuid: "ab0344fe51724f08b8ec8011f569ae1e" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 38615 } } }
I20260808 02:27:40.306926 2208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:27:40.338311 2259 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:27:40.340226 2259 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:27:40.357810 2259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab0344fe51724f08b8ec8011f569ae1e: Generated new TSK 0
I20260808 02:27:40.359241 2259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:27:41.034708 2238 catalog_manager.cc:1259] Loaded cluster ID: 40da4c3cb2db46a091d9017091cf1642
I20260808 02:27:41.034811 2238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: loading cluster ID for follower catalog manager: success
I20260808 02:27:41.037189 2238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: acquiring CA information for follower catalog manager: success
I20260808 02:27:41.039014 2238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b2a02224cb242cba1e2eb51a355e4d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:27:41.122992 2255 catalog_manager.cc:1259] Loaded cluster ID: 40da4c3cb2db46a091d9017091cf1642
I20260808 02:27:41.123129 2255 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: loading cluster ID for follower catalog manager: success
I20260808 02:27:41.125597 2255 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: acquiring CA information for follower catalog manager: success
I20260808 02:27:41.127456 2255 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6d9a04205634b2d9afbbed40651c273: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:27:41.505249 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 1971
I20260808 02:27:41.507184 2035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:41.620205 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 1971
I20260808 02:27:41.648314 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2048
I20260808 02:27:41.649482 2116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:41.759663 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2048
I20260808 02:27:41.786908 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2130
I20260808 02:27:41.788130 2195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:41.898854 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2130
2026-08-08T02:27:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5090 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:27:42 dist-test-slave-pgkr krb5kdc[2280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:27:42 dist-test-slave-pgkr krb5kdc[2280](info): set up 2 sockets
Aug 08 02:27:42 dist-test-slave-pgkr krb5kdc[2280](info): commencing operation
krb5kdc: starting...
W20260808 02:27:43.067699 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.005s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:27:43 dist-test-slave-pgkr krb5kdc[2280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:27:43.366317 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37915
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:43.731573 2294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:43.731866 2294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:43.732036 2294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:43.739392 2294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:43.739497 2294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:43.739595 2294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:43.739703 2294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:43.739789 2294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:43.749763 2294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:43.751169 2294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:43.752556 2294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:43.765306 2299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:43.765306 2300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:43.765352 2302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:43.766389 2294 server_base.cc:1034] running on GCE node
I20260808 02:27:43.767345 2294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:43.768791 2294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:43.770010 2294 hybrid_clock.cc:648] HybridClock initialized: now 1786156063769958 us; error 37 us; skew 500 ppm
I20260808 02:27:43.770411 2294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:43 dist-test-slave-pgkr krb5kdc[2280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156063, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:43.773636 2294 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:27:43.775367 2294 webserver.cc:460] Webserver started at http://127.19.231.62:36919/ using document root <none> and password file <none>
I20260808 02:27:43.776080 2294 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:43.776235 2294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:43.776577 2294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:43.779749 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "25583d4824a9449e8c4bb5d95483fd30"
format_stamp: "Formatted at 2026-08-08 02:27:43 on dist-test-slave-pgkr"
server_key: "81ded0d31c57c486b7cceaeb3935772b"
server_key_iv: "048ed5c70e89b50b63947002d18106c9"
server_key_version: "encryptionkey@0"
I20260808 02:27:43.780536 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "25583d4824a9449e8c4bb5d95483fd30"
format_stamp: "Formatted at 2026-08-08 02:27:43 on dist-test-slave-pgkr"
server_key: "81ded0d31c57c486b7cceaeb3935772b"
server_key_iv: "048ed5c70e89b50b63947002d18106c9"
server_key_version: "encryptionkey@0"
I20260808 02:27:43.785880 2294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 02:27:43.789517 2309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:43.790890 2294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:27:43.791106 2294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "25583d4824a9449e8c4bb5d95483fd30"
format_stamp: "Formatted at 2026-08-08 02:27:43 on dist-test-slave-pgkr"
server_key: "81ded0d31c57c486b7cceaeb3935772b"
server_key_iv: "048ed5c70e89b50b63947002d18106c9"
server_key_version: "encryptionkey@0"
I20260808 02:27:43.791312 2294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:43.836360 2294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:43.841203 2294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:43.841516 2294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:43.861783 2294 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:37915
I20260808 02:27:43.861817 2363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:37915 every 8 connection(s)
I20260808 02:27:43.863360 2294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:27:43.865958 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2294
I20260808 02:27:43.866217 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:27:43.867122 20380 external_mini_cluster.cc:1428] Setting key abf4faf9367deeac9de6c0c1131f5d01
I20260808 02:27:43.868060 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:43.880491 2364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30: Bootstrap starting.
Aug 08 02:27:43 dist-test-slave-pgkr krb5kdc[2280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:43.884547 2364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:43.885674 2364 log.cc:824] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:43.888957 2364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30: No bootstrap required, opened a new log
I20260808 02:27:43.890533 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:43.869079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.19.231.62:37915)
0808 02:27:43.869884 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:43.869896 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:43.869937 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:27:43.871638 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:43.871662 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:43.872062 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:43.872514 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:43.874145 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:43.875557 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:43.876955 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:43.877681 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:43.877914 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:43.882919 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:43.882927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:43.882941 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:43.882987 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:43.886221 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:43.887082 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:43.887089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:43.887098 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:43.887256 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:43.887696 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:43.887700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:43.887702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:43.888226 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:43.888453 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:43.888950 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 02:27:43.889505 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":380,"threads_started":1}
I20260808 02:27:43.895071 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:27:43.898468 2364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25583d4824a9449e8c4bb5d95483fd30" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37915 } }
I20260808 02:27:43.898905 2364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:43.899036 2364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25583d4824a9449e8c4bb5d95483fd30, State: Initialized, Role: FOLLOWER
I20260808 02:27:43.899682 2364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25583d4824a9449e8c4bb5d95483fd30" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37915 } }
I20260808 02:27:43.899888 2364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:27:43.900000 2364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:27:43.900135 2364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:43.902236 2364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25583d4824a9449e8c4bb5d95483fd30" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37915 } }
I20260808 02:27:43.902756 2364 leader_election.cc:302] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25583d4824a9449e8c4bb5d95483fd30; no voters:
I20260808 02:27:43.903287 2364 leader_election.cc:288] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:27:43.903455 2370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:27:43.904181 2370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [term 1 LEADER]: Becoming Leader. State: Replica: 25583d4824a9449e8c4bb5d95483fd30, State: Running, Role: LEADER
I20260808 02:27:43.904707 2370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25583d4824a9449e8c4bb5d95483fd30" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37915 } }
I20260808 02:27:43.905378 2364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:43.907948 2371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25583d4824a9449e8c4bb5d95483fd30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25583d4824a9449e8c4bb5d95483fd30" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37915 } } }
I20260808 02:27:43.907974 2372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25583d4824a9449e8c4bb5d95483fd30. Latest consensus state: current_term: 1 leader_uuid: "25583d4824a9449e8c4bb5d95483fd30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25583d4824a9449e8c4bb5d95483fd30" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37915 } } }
I20260808 02:27:43.908397 2372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:43.908397 2371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30 [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:43.910168 2377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:27:43.914835 2377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:27:43.923527 2377 catalog_manager.cc:1347] Generated new cluster ID: ad3e5d5ab7d24f07b6e95a1f12eda110
I20260808 02:27:43.923650 2377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:27:43.940228 2377 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:27:43.941779 2377 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:27:43.951277 2377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25583d4824a9449e8c4bb5d95483fd30: Generated new TSK 0
I20260808 02:27:43.952200 2377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:27:44.008533 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2294
I20260808 02:27:44.009763 2359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:44.111481 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2294
2026-08-08T02:27:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2208 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:27:44 dist-test-slave-pgkr krb5kdc[2394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:27:44 dist-test-slave-pgkr krb5kdc[2394](info): set up 2 sockets
Aug 08 02:27:44 dist-test-slave-pgkr krb5kdc[2394](info): commencing operation
krb5kdc: starting...
W20260808 02:27:45.269275 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:27:45 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:27:45.572253 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45107
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:45.922482 2407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:45.922871 2407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:45.923075 2407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:45.930502 2407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:45.930665 2407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:45.930788 2407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:45.930895 2407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:45.930967 2407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:45.941105 2407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:45.942564 2407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:45.943996 2407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:45.956229 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:45.956310 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:45.956338 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:45.957170 2407 server_base.cc:1034] running on GCE node
I20260808 02:27:45.958039 2407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:45.959568 2407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:45.960822 2407 hybrid_clock.cc:648] HybridClock initialized: now 1786156065960781 us; error 57 us; skew 500 ppm
I20260808 02:27:45.961203 2407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:45 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156065, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:45.964497 2407 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:27:45.965922 2407 webserver.cc:460] Webserver started at http://127.19.231.62:36683/ using document root <none> and password file <none>
I20260808 02:27:45.966576 2407 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:45.966741 2407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:45.966997 2407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:45.970000 2407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "8521c085b0324fe9acbe4d9dce3a33dc"
format_stamp: "Formatted at 2026-08-08 02:27:45 on dist-test-slave-pgkr"
server_key: "3c51a5eca908f81fb73df4060f80dab5"
server_key_iv: "bef664bd6eec9bb560c862d5394ad88a"
server_key_version: "encryptionkey@0"
I20260808 02:27:45.970813 2407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "8521c085b0324fe9acbe4d9dce3a33dc"
format_stamp: "Formatted at 2026-08-08 02:27:45 on dist-test-slave-pgkr"
server_key: "3c51a5eca908f81fb73df4060f80dab5"
server_key_iv: "bef664bd6eec9bb560c862d5394ad88a"
server_key_version: "encryptionkey@0"
I20260808 02:27:45.976492 2407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:27:45.980022 2423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:45.981408 2407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:27:45.981599 2407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "8521c085b0324fe9acbe4d9dce3a33dc"
format_stamp: "Formatted at 2026-08-08 02:27:45 on dist-test-slave-pgkr"
server_key: "3c51a5eca908f81fb73df4060f80dab5"
server_key_iv: "bef664bd6eec9bb560c862d5394ad88a"
server_key_version: "encryptionkey@0"
I20260808 02:27:45.981803 2407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:45.994460 2407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:45.998440 2407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:45.998786 2407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:46.018493 2407 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:45107
I20260808 02:27:46.018534 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:45107 every 8 connection(s)
I20260808 02:27:46.020030 2407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:27:46.023162 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2407
I20260808 02:27:46.023392 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:27:46.024276 2479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:46.024267 20380 external_mini_cluster.cc:1428] Setting key 167b8fc68322d2359d17de2c25aaf09f
I20260808 02:27:46.039723 2479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Bootstrap starting.
Aug 08 02:27:46 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:46.043092 2479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:46.044159 2479 log.cc:824] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:46.047236 2479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: No bootstrap required, opened a new log
I20260808 02:27:46.049463 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:46.026400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.19.231.62:45107)
0808 02:27:46.027345 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:46.027358 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:46.027402 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:27:46.029273 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:46.029295 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:46.029686 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:46.030029 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:46.031940 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:46.033286 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:46.035257 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:46.035981 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:46.036124 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:46.041420 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:46.041429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:46.041444 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:46.041509 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:46.045162 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:46.045937 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:46.045944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:46.045950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:46.046143 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:46.046549 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:46.046553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:46.046555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:46.047183 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:46.047541 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:46.048000 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 02:27:46.048436 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":447,"threads_started":1}
I20260808 02:27:46.056452 2479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:46.056861 2479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:46.056983 2479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8521c085b0324fe9acbe4d9dce3a33dc, State: Initialized, Role: FOLLOWER
I20260808 02:27:46.057627 2479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:46.057875 2479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:27:46.058046 2479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:27:46.058241 2479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:46.060638 2479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:46.061144 2479 leader_election.cc:302] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8521c085b0324fe9acbe4d9dce3a33dc; no voters:
I20260808 02:27:46.061887 2479 leader_election.cc:288] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:27:46.062042 2484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:27:46.062958 2484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 LEADER]: Becoming Leader. State: Replica: 8521c085b0324fe9acbe4d9dce3a33dc, State: Running, Role: LEADER
I20260808 02:27:46.063496 2484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:46.064126 2479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:46.066701 2486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8521c085b0324fe9acbe4d9dce3a33dc. Latest consensus state: current_term: 1 leader_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } } }
I20260808 02:27:46.067211 2486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:46.067312 2485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } } }
I20260808 02:27:46.067693 2485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:46.069798 2493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:27:46.074272 2493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:27:46.083276 2493 catalog_manager.cc:1347] Generated new cluster ID: 2ccbd78b7cc740a9a0af28d2cd4cba68
I20260808 02:27:46.083388 2493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:27:46.102002 2493 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:27:46.103995 2493 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:27:46.113742 2493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Generated new TSK 0
I20260808 02:27:46.114884 2493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:27:46.182689 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45107
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:46.537470 2506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:46.537796 2506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:46.537964 2506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:46.545302 2506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:46.545463 2506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:46.545667 2506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:27:46.556058 2506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:46.557389 2506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:46.558801 2506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:46.572742 2513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:46.572746 2516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:46.572734 2512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:46.576262 2506 server_base.cc:1034] running on GCE node
I20260808 02:27:46.576881 2506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:46.577790 2506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:46.578990 2506 hybrid_clock.cc:648] HybridClock initialized: now 1786156066578920 us; error 55 us; skew 500 ppm
I20260808 02:27:46.579330 2506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:46 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156066, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:46.582652 2506 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:27:46.584151 2506 webserver.cc:460] Webserver started at http://127.19.231.1:39467/ using document root <none> and password file <none>
I20260808 02:27:46.584918 2506 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:46.585084 2506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:46.585438 2506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:46.588541 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "8610e243fe9e42509c218b61528f147d"
format_stamp: "Formatted at 2026-08-08 02:27:46 on dist-test-slave-pgkr"
server_key: "319cd5057bfd57335abe72e27cd14e8d"
server_key_iv: "24bd3b2dedd4ef31512943523d1ac5bf"
server_key_version: "encryptionkey@0"
I20260808 02:27:46.589344 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "8610e243fe9e42509c218b61528f147d"
format_stamp: "Formatted at 2026-08-08 02:27:46 on dist-test-slave-pgkr"
server_key: "319cd5057bfd57335abe72e27cd14e8d"
server_key_iv: "24bd3b2dedd4ef31512943523d1ac5bf"
server_key_version: "encryptionkey@0"
I20260808 02:27:46.594483 2506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:27:46.597939 2524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:46.599411 2506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:27:46.599614 2506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "8610e243fe9e42509c218b61528f147d"
format_stamp: "Formatted at 2026-08-08 02:27:46 on dist-test-slave-pgkr"
server_key: "319cd5057bfd57335abe72e27cd14e8d"
server_key_iv: "24bd3b2dedd4ef31512943523d1ac5bf"
server_key_version: "encryptionkey@0"
I20260808 02:27:46.599820 2506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:46.615758 2506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:46.620476 2506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:46.620790 2506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:46.621770 2506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:46.623183 2506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:46.623255 2506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:46.623431 2506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:46.623485 2506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:46.657356 2506 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:45747
I20260808 02:27:46.657390 2644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:45747 every 8 connection(s)
I20260808 02:27:46.658856 2506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:27:46.664243 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2506
I20260808 02:27:46.664495 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:27:46.665146 20380 external_mini_cluster.cc:1428] Setting key 1bb6ff2f51d77d19709458c856fb64a7
Aug 08 02:27:46 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156066, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:46.682825 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:46.661618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:59697 (local address 127.19.231.62:45107)
0808 02:27:46.662108 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:46.662112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:46.663144 (+ 1032us) server_negotiation.cc:407] Connection header received
0808 02:27:46.665590 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:46.665596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:46.665702 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:46.665851 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:46.667835 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:46.668688 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:46.671313 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:46.671793 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:46.671899 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:46.676583 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:46.676588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:46.676593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:46.676626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:46.679330 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:46.680185 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:46.680193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:46.680197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:46.680357 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:46.680790 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:46.680804 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:46.680810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:46.681207 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:46.681377 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:46.682034 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 02:27:46.682333 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":271,"threads_started":1}
I20260808 02:27:46.683533 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:46.662060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:59697)
0808 02:27:46.662932 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:46.662981 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:46.665261 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:46.666128 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:46.666141 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:46.666820 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:46.667630 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:46.667666 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:46.668867 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:46.668875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:46.671145 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:46.671164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:46.672039 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:46.672049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:46.672428 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:46.673186 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:46.673221 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:46.676332 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:46.679469 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:46.679481 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:46.679499 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:46.680032 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:46.680470 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:46.680473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:46.680475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:46.680668 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:46.681330 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:46.681348 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:46.681646 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 02:27:46.681876 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 02:27:46.682278 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":315,"threads_started":1}
I20260808 02:27:46.685206 2645 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45107
I20260808 02:27:46.685534 2645 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:46.686497 2645 heartbeater.cc:511] Master 127.19.231.62:45107 requested a full tablet report, sending...
I20260808 02:27:46.688782 2444 ts_manager.cc:194] Registered new tserver with Master: 8610e243fe9e42509c218b61528f147d (127.19.231.1:45747)
I20260808 02:27:46.690258 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:59697
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:27:46.734016 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45107
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:47.077348 2651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:47.077688 2651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:47.077864 2651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:47.085409 2651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:47.085561 2651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:47.085734 2651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:27:47.095603 2651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:47.096935 2651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:47.098323 2651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:47.111660 2656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:47.111621 2657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:47.111697 2659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:47.112775 2651 server_base.cc:1034] running on GCE node
I20260808 02:27:47.113394 2651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:47.114298 2651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:47.115525 2651 hybrid_clock.cc:648] HybridClock initialized: now 1786156067115464 us; error 74 us; skew 500 ppm
I20260808 02:27:47.115830 2651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:47 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156067, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:47.119061 2651 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:27:47.120476 2651 webserver.cc:460] Webserver started at http://127.19.231.2:39487/ using document root <none> and password file <none>
I20260808 02:27:47.121104 2651 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:47.121213 2651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:47.121460 2651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:47.124403 2651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "d19e4a415bfe4852886ddfcefb4a8923"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "b87be77feb60d2cfe801c9277ba6faf7"
server_key_iv: "592ff9af1d4e175f204d74c95b7f0e09"
server_key_version: "encryptionkey@0"
I20260808 02:27:47.125139 2651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "d19e4a415bfe4852886ddfcefb4a8923"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "b87be77feb60d2cfe801c9277ba6faf7"
server_key_iv: "592ff9af1d4e175f204d74c95b7f0e09"
server_key_version: "encryptionkey@0"
I20260808 02:27:47.130189 2651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:27:47.133641 2666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:47.135037 2651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:27:47.135217 2651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "d19e4a415bfe4852886ddfcefb4a8923"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "b87be77feb60d2cfe801c9277ba6faf7"
server_key_iv: "592ff9af1d4e175f204d74c95b7f0e09"
server_key_version: "encryptionkey@0"
I20260808 02:27:47.135396 2651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:47.159366 2651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:47.163925 2651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:47.164225 2651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:47.165369 2651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:47.166880 2651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:47.166982 2651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:47.167128 2651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:47.167209 2651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:47.201037 2651 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:40401
I20260808 02:27:47.201123 2786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:40401 every 8 connection(s)
I20260808 02:27:47.202365 2651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:27:47.203312 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2651
I20260808 02:27:47.203528 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:27:47.204164 20380 external_mini_cluster.cc:1428] Setting key 9251cd55c14af8e5c22be30d518cd0dd
Aug 08 02:27:47 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156067, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:47.223487 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.205387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:56807 (local address 127.19.231.62:45107)
0808 02:27:47.205862 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:47.205866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:47.207060 (+ 1194us) server_negotiation.cc:407] Connection header received
0808 02:27:47.208878 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:47.208884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:47.208968 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:47.209098 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:47.210590 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.211391 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.212938 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.213385 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.213522 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.217109 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:47.217118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:47.217126 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:47.217179 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:47.220098 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:47.221027 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:47.221030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:47.221033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:47.221171 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:47.221698 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:47.221702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:47.221704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:47.221963 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:47.222075 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:47.223025 (+ 950us) server_negotiation.cc:299] Negotiation successful
0808 02:27:47.223207 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":269,"threads_started":1}
I20260808 02:27:47.225246 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.206039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:56807)
0808 02:27:47.206921 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:47.206958 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:47.208620 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:47.209339 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:47.209348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:47.209864 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:47.210424 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.210440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.211539 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.211545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.212787 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.212799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.213611 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.213621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.213866 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.214569 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:47.214591 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:47.216867 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:47.220243 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:47.220255 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:47.220274 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:47.220917 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:47.221278 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:47.221286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:47.221292 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:47.221583 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:47.222090 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:47.222100 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:47.222563 (+ 463us) client_negotiation.cc:769] Sending connection context
0808 02:27:47.222960 (+ 397us) client_negotiation.cc:240] Negotiation successful
0808 02:27:47.223563 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":391,"threads_started":1}
I20260808 02:27:47.226862 2787 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45107
I20260808 02:27:47.227185 2787 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:47.227950 2787 heartbeater.cc:511] Master 127.19.231.62:45107 requested a full tablet report, sending...
I20260808 02:27:47.229492 2444 ts_manager.cc:194] Registered new tserver with Master: d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401)
I20260808 02:27:47.230329 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:56807
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:27:47.272192 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45107
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:47.628906 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:47.629459 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:47.629714 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:47.642154 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:47.642422 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:47.642697 2793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:27:47.658736 2793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:47.661164 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:47.663478 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:47.685379 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:47.685272 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:47.685326 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:47.687325 2793 server_base.cc:1034] running on GCE node
I20260808 02:27:47.688335 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:47.689823 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:47.691167 2793 hybrid_clock.cc:648] HybridClock initialized: now 1786156067691091 us; error 81 us; skew 500 ppm
I20260808 02:27:47.691723 2793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:27:47.694255 2645 heartbeater.cc:503] Master 127.19.231.62:45107 was elected leader, sending a full tablet report...
Aug 08 02:27:47 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156067, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:47.697304 2793 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:27:47.699597 2793 webserver.cc:460] Webserver started at http://127.19.231.3:34621/ using document root <none> and password file <none>
I20260808 02:27:47.700834 2793 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:47.701064 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:47.701545 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:27:47.706960 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "dd714139ef754132a34d37e4fa4f9d3c"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "db4c16a155a1caf2e2df0a3c88f6c22f"
server_key_iv: "ec0b37ed7e8e9794c8f1fc1c77e6b568"
server_key_version: "encryptionkey@0"
I20260808 02:27:47.708348 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "dd714139ef754132a34d37e4fa4f9d3c"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "db4c16a155a1caf2e2df0a3c88f6c22f"
server_key_iv: "ec0b37ed7e8e9794c8f1fc1c77e6b568"
server_key_version: "encryptionkey@0"
I20260808 02:27:47.717409 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260808 02:27:47.723322 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:47.725770 2793 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260808 02:27:47.726034 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "dd714139ef754132a34d37e4fa4f9d3c"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "db4c16a155a1caf2e2df0a3c88f6c22f"
server_key_iv: "ec0b37ed7e8e9794c8f1fc1c77e6b568"
server_key_version: "encryptionkey@0"
I20260808 02:27:47.726362 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:47.748984 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:47.756229 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:47.756702 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:47.758288 2793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:47.760388 2793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:47.760545 2793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:47.760794 2793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:47.760921 2793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:47.815044 2793 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:44941
I20260808 02:27:47.815160 2930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:44941 every 8 connection(s)
I20260808 02:27:47.817211 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:27:47.823196 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2793
I20260808 02:27:47.823498 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:27:47.824162 20380 external_mini_cluster.cc:1428] Setting key f1663c8b7f8be0d8c8f52016a2dce805
Aug 08 02:27:47 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156067, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:47.853114 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.821537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:35839 (local address 127.19.231.62:45107)
0808 02:27:47.822429 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:47.822441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:47.824503 (+ 2062us) server_negotiation.cc:407] Connection header received
0808 02:27:47.828323 (+ 3820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:47.828340 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:47.828488 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:47.828698 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:47.831405 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.832785 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.836009 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.836645 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.836805 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.843196 (+ 6391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:47.843203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:47.843206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:47.843260 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:47.847521 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:47.848654 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:47.848663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:47.848667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:47.848856 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:47.849571 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:47.849577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:47.849581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:47.850067 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:47.850407 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:47.851730 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0808 02:27:47.852166 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":385,"threads_started":1}
I20260808 02:27:47.855020 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.822844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:35839)
0808 02:27:47.824181 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:47.824278 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:47.827917 (+ 3639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:47.829113 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:47.829132 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:47.830002 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:47.831173 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.831205 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.833019 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.833025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.835798 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.835820 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.836961 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.836974 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.837342 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.838755 (+ 1413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:47.838809 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:47.842828 (+ 4019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:47.847758 (+ 4930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:47.847770 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:47.847805 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:47.848456 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:47.849062 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:47.849068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:47.849079 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:47.849387 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:47.850334 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:47.850355 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:47.851066 (+ 711us) client_negotiation.cc:769] Sending connection context
0808 02:27:47.851537 (+ 471us) client_negotiation.cc:240] Negotiation successful
0808 02:27:47.852662 (+ 1125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":539,"threads_started":1}
I20260808 02:27:47.857571 2931 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45107
I20260808 02:27:47.858068 2931 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:47.859375 2931 heartbeater.cc:511] Master 127.19.231.62:45107 requested a full tablet report, sending...
I20260808 02:27:47.861605 2444 ts_manager.cc:194] Registered new tserver with Master: dd714139ef754132a34d37e4fa4f9d3c (127.19.231.3:44941)
I20260808 02:27:47.862991 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:35839
I20260808 02:27:47.871099 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:27:47.900532 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.878858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.19.231.62:45107)
0808 02:27:47.879248 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:47.879260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:47.879906 (+ 646us) server_negotiation.cc:407] Connection header received
0808 02:27:47.880131 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:47.880136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:47.880251 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:47.880391 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:47.882755 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.883866 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.886421 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.887070 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.887228 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.892225 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:47.892231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:47.892235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:47.892289 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:47.896439 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:47.897471 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:47.897475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:47.897477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:47.897658 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:47.898164 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:47.898167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:47.898169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:47.898703 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:47.898964 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:47.899480 (+ 516us) server_negotiation.cc:299] Negotiation successful
0808 02:27:47.899868 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 02:27:47.907061 2444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:27:47.917799 2444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:27:47.966431 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.936587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:40401 (local address 127.0.0.1:35250)
0808 02:27:47.948374 (+ 11787us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:47.948484 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:47.948940 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:47.952144 (+ 3204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:47.952150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:47.952216 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:47.952792 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.952804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.956998 (+ 4194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.957015 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.961977 (+ 4962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.962004 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.964536 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.964545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.965048 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.965132 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 02:27:47.965416 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 02:27:47.965853 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10893,"thread_start_us":18709,"threads_started":1}
I20260808 02:27:47.967813 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.936813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.19.231.2:40401)
0808 02:27:47.947020 (+ 10207us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:47.947046 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:47.949486 (+ 2440us) server_negotiation.cc:407] Connection header received
0808 02:27:47.949723 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:47.949757 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:47.950465 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:47.951731 (+ 1266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:47.953126 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.956691 (+ 3565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.962242 (+ 5551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.964067 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.964389 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.964883 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:47.966987 (+ 2104us) server_negotiation.cc:299] Negotiation successful
0808 02:27:47.967346 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9820,"thread_start_us":9557,"threads_started":1}
I20260808 02:27:47.970996 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.936941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:44941 (local address 127.0.0.1:54506)
0808 02:27:47.955116 (+ 18175us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:47.955149 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:47.955343 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:47.957216 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:47.957223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:47.957278 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:47.958393 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.958423 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.961319 (+ 2896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.961329 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.965488 (+ 4159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.965513 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.969416 (+ 3903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.969422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.969659 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.969707 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:27:47.969873 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 02:27:47.970116 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17917,"mutex_wait_us":1,"spinlock_wait_cycles":7040,"thread_start_us":12268,"threads_started":1}
I20260808 02:27:47.971755 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.937711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.19.231.3:44941)
0808 02:27:47.951666 (+ 13955us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:47.951686 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:47.955608 (+ 3922us) server_negotiation.cc:407] Connection header received
0808 02:27:47.955829 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:47.955870 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:47.956348 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:47.956682 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:47.958798 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.961037 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.965886 (+ 4849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.968777 (+ 2891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.968890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.969204 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:47.969968 (+ 764us) server_negotiation.cc:299] Negotiation successful
0808 02:27:47.970236 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13616,"thread_start_us":13254,"threads_started":1}
I20260808 02:27:47.975026 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.942575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55568)
0808 02:27:47.955799 (+ 13224us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:47.955817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:47.956067 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:47.957470 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:47.957479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:47.957545 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:47.958601 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.958715 (+ 114us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.962044 (+ 3329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.962060 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.968098 (+ 6038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:47.968125 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.973409 (+ 5284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.973425 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:47.973756 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.973921 (+ 165us) client_negotiation.cc:769] Sending connection context
0808 02:27:47.974065 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 02:27:47.974366 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12995,"mutex_wait_us":24,"thread_start_us":11855,"threads_started":1}
I20260808 02:27:47.976254 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:47.944249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.19.231.1:45747)
0808 02:27:47.954197 (+ 9948us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:47.954219 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:47.956329 (+ 2110us) server_negotiation.cc:407] Connection header received
0808 02:27:47.956524 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:47.956538 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:47.956989 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:47.957340 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:47.959044 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.961729 (+ 2685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.968605 (+ 6876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:47.973030 (+ 4425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:47.973335 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:47.973828 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:47.975639 (+ 1811us) server_negotiation.cc:299] Negotiation successful
0808 02:27:47.975870 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9634,"thread_start_us":9351,"threads_started":1}
I20260808 02:27:47.976271 2718 tablet_service.cc:1467] Processing CreateTablet for tablet 138a6e3d10334b798228d2ab754cf6ef (DEFAULT_TABLE table=test-table [id=77caa653e66d47b0b534e19bbf531332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:47.980051 2718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138a6e3d10334b798228d2ab754cf6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:47.980147 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 138a6e3d10334b798228d2ab754cf6ef (DEFAULT_TABLE table=test-table [id=77caa653e66d47b0b534e19bbf531332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:47.983165 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138a6e3d10334b798228d2ab754cf6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:47.983769 2575 tablet_service.cc:1467] Processing CreateTablet for tablet 138a6e3d10334b798228d2ab754cf6ef (DEFAULT_TABLE table=test-table [id=77caa653e66d47b0b534e19bbf531332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:47.986601 2575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138a6e3d10334b798228d2ab754cf6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:48.004065 2950 tablet_bootstrap.cc:492] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Bootstrap starting.
I20260808 02:27:48.012058 2950 tablet_bootstrap.cc:654] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:48.012826 2951 tablet_bootstrap.cc:492] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Bootstrap starting.
I20260808 02:27:48.015105 2950 log.cc:824] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:48.016417 2952 tablet_bootstrap.cc:492] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Bootstrap starting.
I20260808 02:27:48.020897 2951 tablet_bootstrap.cc:654] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:48.022897 2950 tablet_bootstrap.cc:492] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: No bootstrap required, opened a new log
I20260808 02:27:48.023154 2952 tablet_bootstrap.cc:654] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:48.023420 2950 ts_tablet_manager.cc:1397] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20260808 02:27:48.023746 2951 log.cc:824] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:48.025471 2952 log.cc:824] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:48.034672 2952 tablet_bootstrap.cc:492] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: No bootstrap required, opened a new log
I20260808 02:27:48.035198 2952 ts_tablet_manager.cc:1397] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20260808 02:27:48.047922 2951 tablet_bootstrap.cc:492] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: No bootstrap required, opened a new log
I20260808 02:27:48.048442 2951 ts_tablet_manager.cc:1397] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.017s
I20260808 02:27:48.049110 2950 raft_consensus.cc:348] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.050036 2950 raft_consensus.cc:374] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:48.050197 2950 raft_consensus.cc:729] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19e4a415bfe4852886ddfcefb4a8923, State: Initialized, Role: FOLLOWER
I20260808 02:27:48.051445 2950 consensus_queue.cc:260] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.058336 2952 raft_consensus.cc:348] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.059629 2952 raft_consensus.cc:374] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:48.059983 2952 raft_consensus.cc:729] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8610e243fe9e42509c218b61528f147d, State: Initialized, Role: FOLLOWER
I20260808 02:27:48.061765 2952 consensus_queue.cc:260] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.064476 2787 heartbeater.cc:503] Master 127.19.231.62:45107 was elected leader, sending a full tablet report...
I20260808 02:27:48.066552 2950 ts_tablet_manager.cc:1428] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Time spent starting tablet: real 0.043s user 0.038s sys 0.004s
I20260808 02:27:48.068614 2951 raft_consensus.cc:348] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.069603 2951 raft_consensus.cc:374] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:48.069787 2951 raft_consensus.cc:729] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd714139ef754132a34d37e4fa4f9d3c, State: Initialized, Role: FOLLOWER
I20260808 02:27:48.070345 2952 ts_tablet_manager.cc:1428] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Time spent starting tablet: real 0.035s user 0.029s sys 0.004s
I20260808 02:27:48.071527 2951 consensus_queue.cc:260] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.074460 2931 heartbeater.cc:503] Master 127.19.231.62:45107 was elected leader, sending a full tablet report...
I20260808 02:27:48.076642 2951 ts_tablet_manager.cc:1428] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Time spent starting tablet: real 0.028s user 0.026s sys 0.001s
I20260808 02:27:48.081476 2957 raft_consensus.cc:482] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:27:48.081866 2957 raft_consensus.cc:504] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.085640 2957 leader_election.cc:288] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401), dd714139ef754132a34d37e4fa4f9d3c (127.19.231.3:44941)
I20260808 02:27:48.096419 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.085152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:43909 (local address 127.19.231.3:44941)
0808 02:27:48.085570 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:48.085590 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:48.087913 (+ 2323us) server_negotiation.cc:407] Connection header received
0808 02:27:48.088219 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:48.088239 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:48.088394 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:48.088568 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:48.089368 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.090425 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.093377 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.094710 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.094991 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.095173 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:48.095694 (+ 521us) server_negotiation.cc:299] Negotiation successful
0808 02:27:48.095927 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260808 02:27:48.096501 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.085060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:47069 (local address 127.19.231.2:40401)
0808 02:27:48.085449 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:48.085471 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:48.087064 (+ 1593us) server_negotiation.cc:407] Connection header received
0808 02:27:48.087558 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:48.087567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:48.087776 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:48.087968 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:48.089216 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.090366 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.093461 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.094751 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.095110 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.095212 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:48.095784 (+ 572us) server_negotiation.cc:299] Negotiation successful
0808 02:27:48.096028 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260808 02:27:48.096694 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.086355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:44941 (local address 127.19.231.1:43909)
0808 02:27:48.087692 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:48.087741 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:48.087982 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:48.088570 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:48.088574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:48.088593 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:48.089126 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:48.089159 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.090703 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.090707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:48.093156 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:48.093183 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.095112 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.095130 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:48.095458 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.095510 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 02:27:48.095622 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 02:27:48.095870 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":110,"thread_start_us":757,"threads_started":1}
I20260808 02:27:48.097234 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.085025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:40401 (local address 127.19.231.1:47069)
0808 02:27:48.086805 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:48.086879 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:48.087287 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:48.087996 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:48.088007 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:48.088055 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:48.088985 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:48.089005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.090557 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.090575 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:48.093187 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:48.093208 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.095189 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.095199 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:48.095511 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.095557 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:27:48.095733 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 02:27:48.095990 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1381,"mutex_wait_us":134,"thread_start_us":1340,"threads_started":1}
I20260808 02:27:48.098493 2885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a6e3d10334b798228d2ab754cf6ef" candidate_uuid: "8610e243fe9e42509c218b61528f147d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd714139ef754132a34d37e4fa4f9d3c" is_pre_election: true
I20260808 02:27:48.098861 2739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a6e3d10334b798228d2ab754cf6ef" candidate_uuid: "8610e243fe9e42509c218b61528f147d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19e4a415bfe4852886ddfcefb4a8923" is_pre_election: true
I20260808 02:27:48.099398 2885 raft_consensus.cc:2454] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8610e243fe9e42509c218b61528f147d in term 0.
I20260808 02:27:48.099673 2739 raft_consensus.cc:2454] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8610e243fe9e42509c218b61528f147d in term 0.
I20260808 02:27:48.100479 2525 leader_election.cc:302] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8610e243fe9e42509c218b61528f147d, dd714139ef754132a34d37e4fa4f9d3c; no voters:
I20260808 02:27:48.101061 2957 raft_consensus.cc:2781] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:27:48.101346 2957 raft_consensus.cc:482] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:27:48.101529 2957 raft_consensus.cc:3037] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:48.105501 2957 raft_consensus.cc:504] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.106572 2957 leader_election.cc:288] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [CANDIDATE]: Term 1 election: Requested vote from peers d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401), dd714139ef754132a34d37e4fa4f9d3c (127.19.231.3:44941)
I20260808 02:27:48.107300 2739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a6e3d10334b798228d2ab754cf6ef" candidate_uuid: "8610e243fe9e42509c218b61528f147d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19e4a415bfe4852886ddfcefb4a8923"
I20260808 02:27:48.107589 2739 raft_consensus.cc:3037] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:48.107506 2885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138a6e3d10334b798228d2ab754cf6ef" candidate_uuid: "8610e243fe9e42509c218b61528f147d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd714139ef754132a34d37e4fa4f9d3c"
I20260808 02:27:48.107933 2885 raft_consensus.cc:3037] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:48.111402 2739 raft_consensus.cc:2454] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8610e243fe9e42509c218b61528f147d in term 1.
I20260808 02:27:48.112066 2528 leader_election.cc:302] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8610e243fe9e42509c218b61528f147d, d19e4a415bfe4852886ddfcefb4a8923; no voters:
I20260808 02:27:48.112411 2885 raft_consensus.cc:2454] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8610e243fe9e42509c218b61528f147d in term 1.
I20260808 02:27:48.112527 2957 raft_consensus.cc:2781] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:27:48.113373 2957 raft_consensus.cc:686] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 1 LEADER]: Becoming Leader. State: Replica: 8610e243fe9e42509c218b61528f147d, State: Running, Role: LEADER
I20260808 02:27:48.114320 2957 consensus_queue.cc:237] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:48.121857 2443 catalog_manager.cc:5495] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8610e243fe9e42509c218b61528f147d (127.19.231.1). New cstate: current_term: 1 leader_uuid: "8610e243fe9e42509c218b61528f147d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } health_report { overall_health: HEALTHY } } }
W20260808 02:27:48.161922 2646 tablet.cc:2369] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:27:48.176302 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.168039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.19.231.1:45747)
0808 02:27:48.168293 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:48.168305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:48.168331 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:27:48.168466 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:48.168469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:48.168621 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:48.168818 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:27:48.169335 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.170848 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.173072 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.173735 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.173881 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.174733 (+ 852us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:27:48.175571 (+ 838us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:27:48.175734 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:48.175874 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 02:27:48.175974 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 02:27:48.190330 2885 raft_consensus.cc:1261] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 1 FOLLOWER]: Refusing update from remote peer 8610e243fe9e42509c218b61528f147d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:48.190330 2739 raft_consensus.cc:1261] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Refusing update from remote peer 8610e243fe9e42509c218b61528f147d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:48.192167 2962 consensus_queue.cc:1035] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:27:48.192782 2957 consensus_queue.cc:1035] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 02:27:48.205670 2788 tablet.cc:2369] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:27:48.223205 2966 mvcc.cc:205] Tried to move back new op lower bound from 7316095255288680448 to 7316095255003279360. Current Snapshot: MvccSnapshot[applied={T|T < 7316095255288680448}]
I20260808 02:27:48.249724 2443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:27:48.250205 2443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:27:48.257035 2443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Sending DeleteTablet for 3 replicas of tablet 138a6e3d10334b798228d2ab754cf6ef
I20260808 02:27:48.259512 2575 tablet_service.cc:1514] Processing DeleteTablet for tablet 138a6e3d10334b798228d2ab754cf6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:48 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:55568
I20260808 02:27:48.259740 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 138a6e3d10334b798228d2ab754cf6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:48 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:54506
I20260808 02:27:48.259763 2718 tablet_service.cc:1514] Processing DeleteTablet for tablet 138a6e3d10334b798228d2ab754cf6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:48 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:35250
I20260808 02:27:48.262593 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2506
I20260808 02:27:48.272686 2976 tablet_replica.cc:330] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: stopping tablet replica
I20260808 02:27:48.273200 2978 tablet_replica.cc:330] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: stopping tablet replica
I20260808 02:27:48.274173 2976 raft_consensus.cc:2229] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:27:48.275377 2978 raft_consensus.cc:2229] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:27:48.276218 2976 raft_consensus.cc:2258] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:48.276754 2978 raft_consensus.cc:2258] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:48.278507 2977 tablet_replica.cc:330] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: stopping tablet replica
I20260808 02:27:48.279284 2977 raft_consensus.cc:2229] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:27:48.280191 2977 raft_consensus.cc:2258] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:48.283516 2976 ts_tablet_manager.cc:1905] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:48.283818 2978 ts_tablet_manager.cc:1905] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:48.284981 2977 ts_tablet_manager.cc:1905] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:48.300336 2976 ts_tablet_manager.cc:1918] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:48.300657 2976 log.cc:1196] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/138a6e3d10334b798228d2ab754cf6ef
I20260808 02:27:48.301698 2976 ts_tablet_manager.cc:1939] T 138a6e3d10334b798228d2ab754cf6ef P 8610e243fe9e42509c218b61528f147d: Deleting consensus metadata
I20260808 02:27:48.302428 2977 ts_tablet_manager.cc:1918] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:48.302989 2977 log.cc:1196] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/138a6e3d10334b798228d2ab754cf6ef
I20260808 02:27:48.304381 2977 ts_tablet_manager.cc:1939] T 138a6e3d10334b798228d2ab754cf6ef P dd714139ef754132a34d37e4fa4f9d3c: Deleting consensus metadata
I20260808 02:27:48.305604 2426 catalog_manager.cc:4841] TS 8610e243fe9e42509c218b61528f147d (127.19.231.1:45747): tablet 138a6e3d10334b798228d2ab754cf6ef (table test-table [id=77caa653e66d47b0b534e19bbf531332]) successfully deleted
I20260808 02:27:48.308658 2424 catalog_manager.cc:4841] TS dd714139ef754132a34d37e4fa4f9d3c (127.19.231.3:44941): tablet 138a6e3d10334b798228d2ab754cf6ef (table test-table [id=77caa653e66d47b0b534e19bbf531332]) successfully deleted
I20260808 02:27:48.311122 2978 ts_tablet_manager.cc:1918] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:48.311507 2978 log.cc:1196] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/138a6e3d10334b798228d2ab754cf6ef
I20260808 02:27:48.312520 2978 ts_tablet_manager.cc:1939] T 138a6e3d10334b798228d2ab754cf6ef P d19e4a415bfe4852886ddfcefb4a8923: Deleting consensus metadata
Aug 08 02:27:48 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:27:48.316643 2428 catalog_manager.cc:4841] TS d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401): tablet 138a6e3d10334b798228d2ab754cf6ef (table test-table [id=77caa653e66d47b0b534e19bbf531332]) successfully deleted
I20260808 02:27:48.328862 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.264747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.19.231.1:45747)
0808 02:27:48.265066 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:48.265109 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:48.271075 (+ 5966us) server_negotiation.cc:407] Connection header received
0808 02:27:48.273650 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:48.273660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:48.273895 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:48.274171 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:48.288693 (+ 14522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.291526 (+ 2833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.303858 (+ 12332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.305703 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.305981 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.317536 (+ 11555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:48.317563 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:48.317604 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:48.317693 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:48.323531 (+ 5838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:48.324949 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:48.324965 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:48.324968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:48.325221 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:48.325859 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:48.325878 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:48.325880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:48.326464 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:48.327025 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:48.327748 (+ 723us) server_negotiation.cc:299] Negotiation successful
0808 02:27:48.328184 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260808 02:27:48.330017 2638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:48.474267 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2506
I20260808 02:27:48.513998 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2651
Aug 08 02:27:48 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:27:48.541343 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.515275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.19.231.2:40401)
0808 02:27:48.515621 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:48.515636 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:48.515664 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:27:48.515954 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:48.515967 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:48.516143 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:48.516310 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:48.518460 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.520223 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.522484 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.523317 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.523474 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.531266 (+ 7792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:48.531285 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:48.531310 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:48.531397 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:48.536904 (+ 5507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:48.537880 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:48.537896 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:48.537899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:48.538117 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:48.538700 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:48.538707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:48.538712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:48.539474 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:48.539931 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:48.540359 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 02:27:48.540740 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260808 02:27:48.542366 2782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:48.673108 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2651
I20260808 02:27:48.706794 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2793
Aug 08 02:27:48 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:27:48.731822 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:48.708094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.19.231.3:44941)
0808 02:27:48.708859 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:48.708866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:48.708913 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:27:48.709067 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:48.709072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:48.709176 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:48.709318 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:48.711638 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.712808 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.714798 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:48.715427 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:48.715586 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:48.722567 (+ 6981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:48.722596 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:48.722688 (+ 92us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:48.722764 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:48.727214 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:48.728237 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:48.728253 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:48.728256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:48.728469 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:48.729035 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:48.729040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:48.729043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:48.729802 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:48.730201 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:48.730590 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 02:27:48.730981 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":415,"threads_started":1}
I20260808 02:27:48.732889 2926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:48.864698 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2793
I20260808 02:27:48.895076 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2407
I20260808 02:27:48.896334 2474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:49.009512 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2407
I20260808 02:27:49.038235 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45107
--webserver_interface=127.19.231.62
--webserver_port=36683
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:49.390486 2986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:49.390794 2986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:49.390976 2986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:49.398478 2986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:49.398667 2986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:49.398782 2986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:49.398895 2986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:49.398974 2986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:49.409216 2986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=36683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:49.410578 2986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:49.412026 2986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:49.442023 2993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:49.441990 2992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:49.442377 2995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:49.448284 2986 server_base.cc:1034] running on GCE node
I20260808 02:27:49.449501 2986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:49.451552 2986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:49.452827 2986 hybrid_clock.cc:648] HybridClock initialized: now 1786156069452767 us; error 36 us; skew 500 ppm
I20260808 02:27:49.453294 2986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:49 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:49.462791 2986 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:27:49.464632 2986 webserver.cc:460] Webserver started at http://127.19.231.62:36683/ using document root <none> and password file <none>
I20260808 02:27:49.465509 2986 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:49.465694 2986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:49.472155 2986 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:27:49.475481 3005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:49.476824 2986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:27:49.477017 2986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "8521c085b0324fe9acbe4d9dce3a33dc"
format_stamp: "Formatted at 2026-08-08 02:27:45 on dist-test-slave-pgkr"
server_key: "3c51a5eca908f81fb73df4060f80dab5"
server_key_iv: "bef664bd6eec9bb560c862d5394ad88a"
server_key_version: "encryptionkey@0"
I20260808 02:27:49.478125 2986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:49.503405 2986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:49.507997 2986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:49.508313 2986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:49.529284 2986 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:45107
I20260808 02:27:49.529340 3058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:45107 every 8 connection(s)
I20260808 02:27:49.530995 2986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:27:49.538537 3059 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:27:49.539405 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 2986
I20260808 02:27:49.540215 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:45747
--local_ip_for_outbound_sockets=127.19.231.1
--tserver_master_addrs=127.19.231.62:45107
--webserver_port=39467
--webserver_interface=127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:27:49.542234 3059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Bootstrap starting.
I20260808 02:27:49.563271 3059 log.cc:824] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:49.576409 3059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:27:49.576990 3059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Bootstrap complete.
I20260808 02:27:49.587044 3059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:49.587951 3059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8521c085b0324fe9acbe4d9dce3a33dc, State: Initialized, Role: FOLLOWER
I20260808 02:27:49.588639 3059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:49.588903 3059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:27:49.589042 3059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:27:49.589277 3059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:27:49.593564 3059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:49.594141 3059 leader_election.cc:302] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8521c085b0324fe9acbe4d9dce3a33dc; no voters:
I20260808 02:27:49.594868 3059 leader_election.cc:288] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:27:49.595022 3063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:27:49.596341 3063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 LEADER]: Becoming Leader. State: Replica: 8521c085b0324fe9acbe4d9dce3a33dc, State: Running, Role: LEADER
I20260808 02:27:49.596976 3063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:49.597329 3059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:49.599967 3064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } } }
I20260808 02:27:49.599956 3065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8521c085b0324fe9acbe4d9dce3a33dc. Latest consensus state: current_term: 2 leader_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } } }
I20260808 02:27:49.600785 3064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:49.603251 3065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:49.604820 3069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:27:49.615446 3069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:27:49.617583 3069 catalog_manager.cc:1259] Loaded cluster ID: 2ccbd78b7cc740a9a0af28d2cd4cba68
I20260808 02:27:49.617810 3069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:27:49.624311 3069 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 02:27:49.655889 3069 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 02:27:49.668314 3069 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:27:49.670148 3069 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Loaded TSK: 0
I20260808 02:27:49.671784 3069 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:27:49.923722 3061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:49.924077 3061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:49.924230 3061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:49.931792 3061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:49.931955 3061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:49.932132 3061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:27:49.942683 3061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:45747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=39467
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:49.944113 3061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:49.945497 3061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:49.959162 3087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:49.959177 3088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:49.959848 3090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:49.960615 3061 server_base.cc:1034] running on GCE node
I20260808 02:27:49.961246 3061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:49.962214 3061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:49.963488 3061 hybrid_clock.cc:648] HybridClock initialized: now 1786156069963414 us; error 53 us; skew 500 ppm
I20260808 02:27:49.963876 3061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:49 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:49.967347 3061 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:27:49.968931 3061 webserver.cc:460] Webserver started at http://127.19.231.1:39467/ using document root <none> and password file <none>
I20260808 02:27:49.969635 3061 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:49.969795 3061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:49.975569 3061 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260808 02:27:49.978451 3098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:49.979962 3061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:27:49.980159 3061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "8610e243fe9e42509c218b61528f147d"
format_stamp: "Formatted at 2026-08-08 02:27:46 on dist-test-slave-pgkr"
server_key: "319cd5057bfd57335abe72e27cd14e8d"
server_key_iv: "24bd3b2dedd4ef31512943523d1ac5bf"
server_key_version: "encryptionkey@0"
I20260808 02:27:49.981150 3061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:50.007144 3061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:50.011691 3061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:50.012004 3061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:50.013049 3061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:50.014400 3061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:50.014496 3061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:50.014580 3061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:50.014699 3061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 02:27:50.050997 3061 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:45747
I20260808 02:27:50.051106 3214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:45747 every 8 connection(s)
I20260808 02:27:50.052570 3061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:27:50.061970 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 3061
I20260808 02:27:50.062805 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:40401
--local_ip_for_outbound_sockets=127.19.231.2
--tserver_master_addrs=127.19.231.62:45107
--webserver_port=39487
--webserver_interface=127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:27:50 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:50.077227 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:50.055686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:32771 (local address 127.19.231.62:45107)
0808 02:27:50.056325 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:50.056338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:50.057062 (+ 724us) server_negotiation.cc:407] Connection header received
0808 02:27:50.059357 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:50.059372 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:50.059734 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:50.060070 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:50.061567 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.063007 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.064580 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.065293 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.065514 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:50.069101 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:50.069109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:50.069124 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:50.069169 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:50.072146 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:50.072967 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:50.072980 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:50.072985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:50.073152 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:50.073596 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:50.073602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:50.073605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:50.074405 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:50.074612 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:50.075522 (+ 910us) server_negotiation.cc:299] Negotiation successful
0808 02:27:50.075952 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":300,"threads_started":1}
I20260808 02:27:50.077232 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:50.055948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:32771)
0808 02:27:50.056880 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:50.056926 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:50.058788 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:50.060207 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:50.060215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:50.060751 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:50.061310 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:50.061348 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.063211 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.063216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:50.064427 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:50.064439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.065542 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.065550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:50.065839 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:50.066545 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:50.066569 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:50.068820 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:50.072319 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:50.072331 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:50.072346 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:50.072825 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:50.073276 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:50.073280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:50.073284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:50.073479 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:50.074556 (+ 1077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:50.074564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:50.074937 (+ 373us) client_negotiation.cc:769] Sending connection context
0808 02:27:50.075281 (+ 344us) client_negotiation.cc:240] Negotiation successful
0808 02:27:50.075857 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":391,"threads_started":1}
I20260808 02:27:50.080731 3215 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45107
I20260808 02:27:50.081048 3215 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:50.081825 3215 heartbeater.cc:511] Master 127.19.231.62:45107 requested a full tablet report, sending...
I20260808 02:27:50.084474 3023 ts_manager.cc:194] Registered new tserver with Master: 8610e243fe9e42509c218b61528f147d (127.19.231.1:45747)
I20260808 02:27:50.086938 3023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:32771
W20260808 02:27:50.422344 3219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:50.422673 3219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:50.422802 3219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:50.430006 3219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:50.430119 3219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:50.430240 3219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:27:50.440426 3219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:40401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=39487
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:50.441754 3219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:50.443171 3219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:50.456817 3224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:50.456771 3228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:50.456864 3226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:50.457739 3219 server_base.cc:1034] running on GCE node
I20260808 02:27:50.458335 3219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:50.459375 3219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:50.460582 3219 hybrid_clock.cc:648] HybridClock initialized: now 1786156070460532 us; error 27 us; skew 500 ppm
I20260808 02:27:50.460934 3219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:50 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156070, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:50.464843 3219 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:27:50.466670 3219 webserver.cc:460] Webserver started at http://127.19.231.2:39487/ using document root <none> and password file <none>
I20260808 02:27:50.467427 3219 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:50.467599 3219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:50.473295 3219 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 02:27:50.476473 3236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:50.477912 3219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:27:50.478097 3219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "d19e4a415bfe4852886ddfcefb4a8923"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "b87be77feb60d2cfe801c9277ba6faf7"
server_key_iv: "592ff9af1d4e175f204d74c95b7f0e09"
server_key_version: "encryptionkey@0"
I20260808 02:27:50.479220 3219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:50.501140 3219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:50.505329 3219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:50.505632 3219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:50.506883 3219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:50.508630 3219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:50.508801 3219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:50.508965 3219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:50.509091 3219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:50.547256 3219 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:40401
I20260808 02:27:50.547338 3353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:40401 every 8 connection(s)
I20260808 02:27:50.549007 3219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:27:50.558190 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 3219
I20260808 02:27:50.559046 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:44941
--local_ip_for_outbound_sockets=127.19.231.3
--tserver_master_addrs=127.19.231.62:45107
--webserver_port=34621
--webserver_interface=127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:27:50 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156070, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:50.569824 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:50.551753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:34243 (local address 127.19.231.62:45107)
0808 02:27:50.551948 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:50.551954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:50.553199 (+ 1245us) server_negotiation.cc:407] Connection header received
0808 02:27:50.555099 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:50.555105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:50.555231 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:50.555350 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:50.556897 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.557594 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.559462 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.559897 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.560052 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:50.563770 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:50.563775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:50.563779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:50.563813 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:50.566705 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:50.567467 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:50.567473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:50.567476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:50.567613 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:50.568069 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:50.568075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:50.568077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:50.568353 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:50.568516 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:50.569233 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 02:27:50.569462 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:27:50.570931 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:50.552270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:34243)
0808 02:27:50.553018 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:50.553074 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:50.554815 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:50.555611 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:50.555620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:50.556136 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:50.556714 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:50.556730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.557792 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.557796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:50.559317 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:50.559329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:50.560150 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:50.560158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:50.560392 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:50.561162 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:50.561192 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:50.563536 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:50.566856 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:50.566865 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:50.566881 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:50.567332 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:50.567736 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:50.567741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:50.567745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:50.567938 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:50.568490 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:50.568498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:50.568844 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 02:27:50.569121 (+ 277us) client_negotiation.cc:240] Negotiation successful
0808 02:27:50.569704 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":311,"threads_started":1}
I20260808 02:27:50.572535 3354 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45107
I20260808 02:27:50.572896 3354 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:50.573922 3354 heartbeater.cc:511] Master 127.19.231.62:45107 requested a full tablet report, sending...
I20260808 02:27:50.575881 3023 ts_manager.cc:194] Registered new tserver with Master: d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401)
I20260808 02:27:50.577260 3023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:34243
W20260808 02:27:50.950232 3357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:50.950572 3357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:50.950766 3357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:50.959519 3357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:50.959654 3357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:50.959790 3357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:27:50.970798 3357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:44941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=34621
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:50.972191 3357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:50.973592 3357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:50.988606 3362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:50.988690 3365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:50.988574 3363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:50.989562 3357 server_base.cc:1034] running on GCE node
I20260808 02:27:50.990250 3357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:50.991351 3357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:50.992687 3357 hybrid_clock.cc:648] HybridClock initialized: now 1786156070992584 us; error 84 us; skew 500 ppm
I20260808 02:27:50.993124 3357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:50 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156070, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:50.997054 3357 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:27:50.998557 3357 webserver.cc:460] Webserver started at http://127.19.231.3:34621/ using document root <none> and password file <none>
I20260808 02:27:50.999341 3357 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:50.999461 3357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:51.005252 3357 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 02:27:51.008455 3372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:51.010030 3357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:27:51.010241 3357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "dd714139ef754132a34d37e4fa4f9d3c"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "db4c16a155a1caf2e2df0a3c88f6c22f"
server_key_iv: "ec0b37ed7e8e9794c8f1fc1c77e6b568"
server_key_version: "encryptionkey@0"
I20260808 02:27:51.011442 3357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:51.026109 3357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:51.030488 3357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:51.030817 3357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:51.031798 3357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:51.033190 3357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:51.033317 3357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:51.033457 3357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:51.033547 3357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:51.071694 3357 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:44941
I20260808 02:27:51.071790 3496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:44941 every 8 connection(s)
I20260808 02:27:51.073356 3357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:27:51.074398 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 3357
Aug 08 02:27:51 dist-test-slave-pgkr krb5kdc[2394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156070, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:27:51.090214 3215 heartbeater.cc:503] Master 127.19.231.62:45107 was elected leader, sending a full tablet report...
I20260808 02:27:51.090651 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.075531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.19.231.62:45107)
0808 02:27:51.076374 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.076378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.076397 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:27:51.076485 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.076488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.076549 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.076687 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:51.078461 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.079325 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.080715 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.081178 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.081317 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.084527 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:51.084533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:51.084538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:51.084579 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:51.087747 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.088444 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.088449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.088453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.088591 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.088983 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.088988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.088992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.089263 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:51.089551 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.089860 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.090089 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":620,"threads_started":1}
I20260808 02:27:51.097680 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.076873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:36177 (local address 127.19.231.62:45107)
0808 02:27:51.077431 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.077437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.078694 (+ 1257us) server_negotiation.cc:407] Connection header received
0808 02:27:51.080751 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.080757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.080856 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.080984 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:51.082984 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.083935 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.086076 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.086648 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.086804 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.090850 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:51.090856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:51.090860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:51.090896 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:51.094304 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.095292 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.095297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.095300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.095455 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.095962 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.095967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.095969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.096279 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:51.096413 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.097219 (+ 806us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.097380 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":276,"threads_started":1}
I20260808 02:27:51.098767 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.077558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:36177)
0808 02:27:51.078469 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.078531 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:51.080456 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:51.081281 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:51.081335 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:51.081960 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:27:51.082797 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.082830 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.084086 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.084092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.085911 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.085925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.086911 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.086922 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.087194 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.087836 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:27:51.087860 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:27:51.090511 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:27:51.094444 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:51.094451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:51.094465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:51.095122 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:51.095624 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:27:51.095628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:27:51.095631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:27:51.095829 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:27:51.096410 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:27:51.096416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:27:51.096790 (+ 374us) client_negotiation.cc:769] Sending connection context
0808 02:27:51.097042 (+ 252us) client_negotiation.cc:240] Negotiation successful
0808 02:27:51.097524 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":377,"threads_started":1}
I20260808 02:27:51.100211 3497 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45107
I20260808 02:27:51.100517 3497 heartbeater.cc:464] Registering TS with master...
I20260808 02:27:51.101279 3497 heartbeater.cc:511] Master 127.19.231.62:45107 requested a full tablet report, sending...
I20260808 02:27:51.102751 3022 ts_manager.cc:194] Registered new tserver with Master: dd714139ef754132a34d37e4fa4f9d3c (127.19.231.3:44941)
I20260808 02:27:51.103607 3022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:36177
I20260808 02:27:51.106542 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:27:51.114058 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.107489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.19.231.62:45107)
0808 02:27:51.107642 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.107646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.108176 (+ 530us) server_negotiation.cc:407] Connection header received
0808 02:27:51.108403 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.108408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.108590 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.108744 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:27:51.109157 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.110002 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.111985 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.112433 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.112536 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.113083 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:27:51.113565 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:27:51.113678 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.113807 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.113870 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 02:27:51.115020 3022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:27:51.122967 3022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:27:51.150216 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.136210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.19.231.2:40401)
0808 02:27:51.136825 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.136837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.139049 (+ 2212us) server_negotiation.cc:407] Connection header received
0808 02:27:51.139629 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.139642 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.139950 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.140212 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:51.141456 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.143717 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.147708 (+ 3991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.149072 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.149241 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.149438 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.149747 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.149916 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":342,"threads_started":1}
I20260808 02:27:51.150336 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.135725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:40401 (local address 127.0.0.1:35264)
0808 02:27:51.138915 (+ 3190us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.138937 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:51.139359 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:51.140599 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:51.140607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:51.140654 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:51.141212 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.141230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.143993 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.144003 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.147463 (+ 3460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.147495 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.149286 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.149293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.149481 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.149537 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 02:27:51.149676 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 02:27:51.149859 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3034,"mutex_wait_us":29,"thread_start_us":1336,"threads_started":1}
I20260808 02:27:51.151476 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.135697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:44941 (local address 127.0.0.1:54526)
0808 02:27:51.137548 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.137600 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:51.137800 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:51.140035 (+ 2235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:51.140046 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:51.140088 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:51.140654 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.140668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.144060 (+ 3392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.144068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.148296 (+ 4228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.148311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.150090 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.150098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.150275 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.150820 (+ 545us) client_negotiation.cc:769] Sending connection context
0808 02:27:51.150915 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:27:51.151111 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"mutex_wait_us":59,"thread_start_us":2864,"threads_started":1}
I20260808 02:27:51.151513 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.135769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.19.231.3:44941)
0808 02:27:51.139088 (+ 3319us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.139097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.139132 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:27:51.139253 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.139264 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.139750 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.140929 (+ 1179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:51.141263 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.143854 (+ 2591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.148511 (+ 4657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.149857 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.150732 (+ 875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.150960 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.151028 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.151197 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2577,"thread_start_us":2393,"threads_started":1}
I20260808 02:27:51.154798 3289 tablet_service.cc:1467] Processing CreateTablet for tablet e95838de27b64686b23de16e2d866b83 (DEFAULT_TABLE table=test-table [id=40e87dd5712c47b6b41feac602c07901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:51.154989 3429 tablet_service.cc:1467] Processing CreateTablet for tablet e95838de27b64686b23de16e2d866b83 (DEFAULT_TABLE table=test-table [id=40e87dd5712c47b6b41feac602c07901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:51.156219 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.139283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.19.231.1:45747)
0808 02:27:51.144910 (+ 5627us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.144920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.147545 (+ 2625us) server_negotiation.cc:407] Connection header received
0808 02:27:51.147752 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.147761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.148025 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.148238 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:51.148861 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.150299 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.153151 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.154559 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.154726 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.154949 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.155711 (+ 762us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.155869 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5453,"thread_start_us":5276,"threads_started":1}
I20260808 02:27:51.156229 3429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95838de27b64686b23de16e2d866b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:51.156319 3289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95838de27b64686b23de16e2d866b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:51.156725 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.139439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55598)
0808 02:27:51.147402 (+ 7963us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.147424 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:51.147560 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:51.148239 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:51.148245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:51.148270 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:51.148660 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.148671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.150472 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.150477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.152950 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.152993 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.155101 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.155108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.155267 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.155297 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 02:27:51.155390 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 02:27:51.156412 (+ 1022us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7802,"thread_start_us":7180,"threads_started":1}
I20260808 02:27:51.159838 3150 tablet_service.cc:1467] Processing CreateTablet for tablet e95838de27b64686b23de16e2d866b83 (DEFAULT_TABLE table=test-table [id=40e87dd5712c47b6b41feac602c07901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:27:51.161017 3150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95838de27b64686b23de16e2d866b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:27:51.172194 3515 tablet_bootstrap.cc:492] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Bootstrap starting.
I20260808 02:27:51.174559 3516 tablet_bootstrap.cc:492] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Bootstrap starting.
I20260808 02:27:51.177026 3515 tablet_bootstrap.cc:654] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:51.177917 3517 tablet_bootstrap.cc:492] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Bootstrap starting.
I20260808 02:27:51.178539 3515 log.cc:824] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:51.178810 3516 tablet_bootstrap.cc:654] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:51.180068 3516 log.cc:824] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:51.181993 3517 tablet_bootstrap.cc:654] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Neither blocks nor log segments found. Creating new log.
I20260808 02:27:51.182974 3515 tablet_bootstrap.cc:492] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: No bootstrap required, opened a new log
I20260808 02:27:51.183270 3515 ts_tablet_manager.cc:1397] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 02:27:51.183256 3517 log.cc:824] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:51.183857 3516 tablet_bootstrap.cc:492] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: No bootstrap required, opened a new log
I20260808 02:27:51.184230 3516 ts_tablet_manager.cc:1397] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260808 02:27:51.186874 3517 tablet_bootstrap.cc:492] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: No bootstrap required, opened a new log
I20260808 02:27:51.187114 3517 ts_tablet_manager.cc:1397] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 02:27:51.191668 3516 raft_consensus.cc:348] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.192027 3516 raft_consensus.cc:374] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:51.192168 3516 raft_consensus.cc:729] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8610e243fe9e42509c218b61528f147d, State: Initialized, Role: FOLLOWER
I20260808 02:27:51.192800 3516 consensus_queue.cc:260] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.194553 3516 ts_tablet_manager.cc:1428] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260808 02:27:51.195403 3515 raft_consensus.cc:348] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.195858 3515 raft_consensus.cc:374] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:51.195961 3515 raft_consensus.cc:729] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19e4a415bfe4852886ddfcefb4a8923, State: Initialized, Role: FOLLOWER
I20260808 02:27:51.196597 3515 consensus_queue.cc:260] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.197798 3517 raft_consensus.cc:348] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.198342 3517 raft_consensus.cc:374] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:27:51.198464 3517 raft_consensus.cc:729] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd714139ef754132a34d37e4fa4f9d3c, State: Initialized, Role: FOLLOWER
I20260808 02:27:51.199398 3517 consensus_queue.cc:260] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.203365 3497 heartbeater.cc:503] Master 127.19.231.62:45107 was elected leader, sending a full tablet report...
I20260808 02:27:51.203804 3354 heartbeater.cc:503] Master 127.19.231.62:45107 was elected leader, sending a full tablet report...
I20260808 02:27:51.204267 3517 ts_tablet_manager.cc:1428] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260808 02:27:51.205830 3515 ts_tablet_manager.cc:1428] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Time spent starting tablet: real 0.022s user 0.011s sys 0.008s
I20260808 02:27:51.225749 3526 raft_consensus.cc:482] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:27:51.226073 3526 raft_consensus.cc:504] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.227916 3526 leader_election.cc:288] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401), 8610e243fe9e42509c218b61528f147d (127.19.231.1:45747)
I20260808 02:27:51.234496 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.228257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.19.231.3:35841)
0808 02:27:51.228484 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.228506 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:51.228660 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:51.229182 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:51.229187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:51.229234 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:51.229457 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.229468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.230891 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.230897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.232388 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.232400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.233735 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.233743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.233933 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.233972 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:27:51.234069 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 02:27:51.234198 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 02:27:51.234957 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.228717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:35841 (local address 127.19.231.1:45747)
0808 02:27:51.228904 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.228908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.228924 (+ 16us) server_negotiation.cc:407] Connection header received
0808 02:27:51.228988 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.228992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.229049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.229180 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:51.229622 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.230771 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.232553 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.233584 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.233704 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.233755 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.234312 (+ 557us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.234435 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:27:51.235328 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.228398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:40829 (local address 127.19.231.2:40401)
0808 02:27:51.228615 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.228621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.229339 (+ 718us) server_negotiation.cc:407] Connection header received
0808 02:27:51.229499 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.229507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.229594 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.229676 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:27:51.230335 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.231246 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.233194 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.234180 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.234311 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.234383 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.234953 (+ 570us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.235081 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:27:51.235435 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.228266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:40401 (local address 127.19.231.3:40829)
0808 02:27:51.229240 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.229254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:27:51.229370 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:27:51.229814 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:27:51.229819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:27:51.229845 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:27:51.230186 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.230198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.231391 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.231395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.233020 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:27:51.233034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.234528 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.234535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:27:51.234734 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.234770 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:27:51.234924 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 02:27:51.235047 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":254,"thread_start_us":420,"threads_started":1}
I20260808 02:27:51.236325 3170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95838de27b64686b23de16e2d866b83" candidate_uuid: "dd714139ef754132a34d37e4fa4f9d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8610e243fe9e42509c218b61528f147d" is_pre_election: true
I20260808 02:27:51.236428 3309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95838de27b64686b23de16e2d866b83" candidate_uuid: "dd714139ef754132a34d37e4fa4f9d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19e4a415bfe4852886ddfcefb4a8923" is_pre_election: true
I20260808 02:27:51.236869 3170 raft_consensus.cc:2454] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd714139ef754132a34d37e4fa4f9d3c in term 0.
I20260808 02:27:51.236934 3309 raft_consensus.cc:2454] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd714139ef754132a34d37e4fa4f9d3c in term 0.
I20260808 02:27:51.237672 3375 leader_election.cc:302] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8610e243fe9e42509c218b61528f147d, dd714139ef754132a34d37e4fa4f9d3c; no voters:
I20260808 02:27:51.238135 3526 raft_consensus.cc:2781] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:27:51.238293 3526 raft_consensus.cc:482] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:27:51.238400 3526 raft_consensus.cc:3037] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:51.240715 3526 raft_consensus.cc:504] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.241341 3526 leader_election.cc:288] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [CANDIDATE]: Term 1 election: Requested vote from peers d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401), 8610e243fe9e42509c218b61528f147d (127.19.231.1:45747)
I20260808 02:27:51.241746 3309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95838de27b64686b23de16e2d866b83" candidate_uuid: "dd714139ef754132a34d37e4fa4f9d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19e4a415bfe4852886ddfcefb4a8923"
I20260808 02:27:51.241767 3170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95838de27b64686b23de16e2d866b83" candidate_uuid: "dd714139ef754132a34d37e4fa4f9d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8610e243fe9e42509c218b61528f147d"
I20260808 02:27:51.241940 3309 raft_consensus.cc:3037] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:51.241951 3170 raft_consensus.cc:3037] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:27:51.244246 3170 raft_consensus.cc:2454] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd714139ef754132a34d37e4fa4f9d3c in term 1.
I20260808 02:27:51.244257 3309 raft_consensus.cc:2454] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd714139ef754132a34d37e4fa4f9d3c in term 1.
I20260808 02:27:51.244637 3375 leader_election.cc:302] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8610e243fe9e42509c218b61528f147d, dd714139ef754132a34d37e4fa4f9d3c; no voters:
I20260808 02:27:51.244966 3526 raft_consensus.cc:2781] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:27:51.249387 3526 raft_consensus.cc:686] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 1 LEADER]: Becoming Leader. State: Replica: dd714139ef754132a34d37e4fa4f9d3c, State: Running, Role: LEADER
I20260808 02:27:51.249960 3526 consensus_queue.cc:237] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } }
I20260808 02:27:51.255707 3022 catalog_manager.cc:5495] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c reported cstate change: term changed from 0 to 1, leader changed from <none> to dd714139ef754132a34d37e4fa4f9d3c (127.19.231.3). New cstate: current_term: 1 leader_uuid: "dd714139ef754132a34d37e4fa4f9d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd714139ef754132a34d37e4fa4f9d3c" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 44941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 } health_report { overall_health: UNKNOWN } } }
I20260808 02:27:51.271364 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.266086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.19.231.3:44941)
0808 02:27:51.266263 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.266268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.266441 (+ 173us) server_negotiation.cc:407] Connection header received
0808 02:27:51.266593 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.266597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.266744 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.266866 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:27:51.267204 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.267902 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.269287 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.269744 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.269836 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.270298 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:27:51.270909 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:27:51.271016 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.271098 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.271170 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:27:51.279811 3170 raft_consensus.cc:1261] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 1 FOLLOWER]: Refusing update from remote peer dd714139ef754132a34d37e4fa4f9d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:51.279796 3309 raft_consensus.cc:1261] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Refusing update from remote peer dd714139ef754132a34d37e4fa4f9d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:27:51.280979 3528 consensus_queue.cc:1035] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19e4a415bfe4852886ddfcefb4a8923" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:27:51.281368 3526 consensus_queue.cc:1035] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8610e243fe9e42509c218b61528f147d" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:27:51.309352 3022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:27:51.309635 3022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:27:51.313136 3022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Sending DeleteTablet for 3 replicas of tablet e95838de27b64686b23de16e2d866b83
I20260808 02:27:51.314409 3289 tablet_service.cc:1514] Processing DeleteTablet for tablet e95838de27b64686b23de16e2d866b83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:51 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:35264
I20260808 02:27:51.314507 3150 tablet_service.cc:1514] Processing DeleteTablet for tablet e95838de27b64686b23de16e2d866b83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:51 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:55598
I20260808 02:27:51.314504 3429 tablet_service.cc:1514] Processing DeleteTablet for tablet e95838de27b64686b23de16e2d866b83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:27:51 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:54526
I20260808 02:27:51.316152 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 3061
I20260808 02:27:51.326872 3543 tablet_replica.cc:330] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: stopping tablet replica
I20260808 02:27:51.327086 3542 tablet_replica.cc:330] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: stopping tablet replica
I20260808 02:27:51.327325 3543 raft_consensus.cc:2229] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:27:51.327447 3542 raft_consensus.cc:2229] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:27:51.327965 3542 raft_consensus.cc:2258] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:51.327946 3543 raft_consensus.cc:2258] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:51.328715 3541 tablet_replica.cc:330] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: stopping tablet replica
I20260808 02:27:51.329130 3541 raft_consensus.cc:2229] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:27:51.329670 3541 raft_consensus.cc:2258] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:27:51.330081 3543 ts_tablet_manager.cc:1905] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:51.331707 3542 ts_tablet_manager.cc:1905] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:51.332682 3541 ts_tablet_manager.cc:1905] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:51.337579 3543 ts_tablet_manager.cc:1918] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:51.337754 3543 log.cc:1196] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/e95838de27b64686b23de16e2d866b83
I20260808 02:27:51.338322 3543 ts_tablet_manager.cc:1939] T e95838de27b64686b23de16e2d866b83 P dd714139ef754132a34d37e4fa4f9d3c: Deleting consensus metadata
I20260808 02:27:51.338366 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.317258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.19.231.1:45747)
0808 02:27:51.317477 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.317486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.317510 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:27:51.317580 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.317588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.317717 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.317844 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:51.320545 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.322087 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.324709 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.325673 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.325848 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.329878 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:51.329891 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:51.329904 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:51.329934 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:51.333988 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.334930 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.334939 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.334943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.335092 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.335566 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.335571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.335575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.336041 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:51.336427 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.337622 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.337975 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:27:51.339200 3210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:51.339262 3542 ts_tablet_manager.cc:1918] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:51.339416 3542 log.cc:1196] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/e95838de27b64686b23de16e2d866b83
I20260808 02:27:51.340152 3006 catalog_manager.cc:4841] TS dd714139ef754132a34d37e4fa4f9d3c (127.19.231.3:44941): tablet e95838de27b64686b23de16e2d866b83 (table test-table [id=40e87dd5712c47b6b41feac602c07901]) successfully deleted
I20260808 02:27:51.340924 3541 ts_tablet_manager.cc:1918] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:51.341130 3541 log.cc:1196] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/e95838de27b64686b23de16e2d866b83
I20260808 02:27:51.341840 3541 ts_tablet_manager.cc:1939] T e95838de27b64686b23de16e2d866b83 P d19e4a415bfe4852886ddfcefb4a8923: Deleting consensus metadata
I20260808 02:27:51.344188 3010 catalog_manager.cc:4841] TS d19e4a415bfe4852886ddfcefb4a8923 (127.19.231.2:40401): tablet e95838de27b64686b23de16e2d866b83 (table test-table [id=40e87dd5712c47b6b41feac602c07901]) successfully deleted
I20260808 02:27:51.436803 3542 ts_tablet_manager.cc:1939] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Deleting consensus metadata
I20260808 02:27:51.436862 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 3061
W20260808 02:27:51.461195 3009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.231.1:45747 (error 108)
I20260808 02:27:51.461447 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 3219
I20260808 02:27:51.463418 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.462909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55612)
0808 02:27:51.463100 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.463235 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 02:27:51.463685 3009 catalog_manager.cc:4584] TS 8610e243fe9e42509c218b61528f147d (127.19.231.1:45747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e95838de27b64686b23de16e2d866b83: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
I20260808 02:27:51.478658 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.462446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.19.231.2:40401)
0808 02:27:51.462657 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.462663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.462724 (+ 61us) server_negotiation.cc:407] Connection header received
0808 02:27:51.462965 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.462972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.463068 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.463181 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:51.465010 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.465750 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.467845 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.468510 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.468624 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.472086 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:51.472105 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:51.472125 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:51.472169 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:51.475738 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.476472 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.476485 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.476488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.476616 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.477022 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.477027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.477032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.477402 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:51.477699 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.478045 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.478287 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:27:51.479496 3349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:51.517484 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.516907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55622)
0808 02:27:51.517156 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.517275 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 02:27:51.518697 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.518236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55632)
0808 02:27:51.518400 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.518491 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 02:27:51.579391 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.578814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55634)
0808 02:27:51.579067 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.579177 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 02:27:51.580722 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.580270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55640)
0808 02:27:51.580458 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.580559 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 02:27:51.584779 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 3219
I20260808 02:27:51.614317 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 3357
I20260808 02:27:51.629422 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.615259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.19.231.3:44941)
0808 02:27:51.615465 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 02:27:51.615471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:27:51.615493 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:27:51.615671 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:27:51.615676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:27:51.615773 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:27:51.615889 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:27:51.617361 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.618172 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.619599 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:27:51.620088 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:27:51.620262 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:27:51.623097 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:27:51.623111 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:27:51.623132 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:27:51.623171 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:27:51.626270 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.627122 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.627130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.627132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.627305 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:27:51.627717 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:27:51.627721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:27:51.627723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:27:51.628078 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:27:51.628453 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0808 02:27:51.628749 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 02:27:51.629002 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:27:51.630163 3492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:51.661397 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.660928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55656)
0808 02:27:51.661135 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.661240 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 02:27:51.662499 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.662091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55662)
0808 02:27:51.662319 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.662388 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 02:27:51.756083 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 3357
I20260808 02:27:51.783844 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 2986
I20260808 02:27:51.785127 3054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:27:51.904763 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 2986
I20260808 02:27:51.904906 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:51.904226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:45747 (local address 127.0.0.1:55672)
0808 02:27:51.904465 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:51.904561 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 02:27:51.933231 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45107
--webserver_interface=127.19.231.62
--webserver_port=36683
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:52.281427 3550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:52.281714 3550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:52.281804 3550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:52.289002 3550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:27:52.289108 3550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:52.289162 3550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:52.289237 3550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:27:52.289283 3550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:27:52.299132 3550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=36683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:52.300540 3550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:52.301878 3550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:52.313973 3555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:52.313916 3556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:52.313884 3558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:52.315208 3550 server_base.cc:1034] running on GCE node
I20260808 02:27:52.316012 3550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:52.317461 3550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:52.318715 3550 hybrid_clock.cc:648] HybridClock initialized: now 1786156072318646 us; error 51 us; skew 500 ppm
I20260808 02:27:52.319080 3550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:52 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156072, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:52.322319 3550 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:27:52.323840 3550 webserver.cc:460] Webserver started at http://127.19.231.62:36683/ using document root <none> and password file <none>
I20260808 02:27:52.324493 3550 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:52.324600 3550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:52.330133 3550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:27:52.333293 3566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:52.334854 3550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:27:52.335070 3550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "8521c085b0324fe9acbe4d9dce3a33dc"
format_stamp: "Formatted at 2026-08-08 02:27:45 on dist-test-slave-pgkr"
server_key: "3c51a5eca908f81fb73df4060f80dab5"
server_key_iv: "bef664bd6eec9bb560c862d5394ad88a"
server_key_version: "encryptionkey@0"
I20260808 02:27:52.336174 3550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:52.355199 3550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:52.359436 3550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:52.359768 3550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:52.379498 3550 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:45107
I20260808 02:27:52.379537 3623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:45107 every 8 connection(s)
I20260808 02:27:52.381340 3550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:27:52.383572 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 3550
I20260808 02:27:52.384459 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:45747
--local_ip_for_outbound_sockets=127.19.231.1
--tserver_master_addrs=127.19.231.62:45107
--webserver_port=39467
--webserver_interface=127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:27:52.388610 3624 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:27:52.391928 3624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Bootstrap starting.
I20260808 02:27:52.413499 3624 log.cc:824] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:27:52.429487 3624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:27:52.430050 3624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc: Bootstrap complete.
I20260808 02:27:52.439476 3624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:52.440616 3624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8521c085b0324fe9acbe4d9dce3a33dc, State: Initialized, Role: FOLLOWER
I20260808 02:27:52.441331 3624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:52.441536 3624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:27:52.441694 3624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:27:52.441886 3624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:27:52.445775 3624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:52.446285 3624 leader_election.cc:302] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8521c085b0324fe9acbe4d9dce3a33dc; no voters:
I20260808 02:27:52.447028 3624 leader_election.cc:288] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:27:52.447172 3628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:27:52.448758 3628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [term 3 LEADER]: Becoming Leader. State: Replica: 8521c085b0324fe9acbe4d9dce3a33dc, State: Running, Role: LEADER
I20260808 02:27:52.449321 3628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } }
I20260808 02:27:52.449625 3624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:27:52.456398 3629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } } }
I20260808 02:27:52.456427 3630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8521c085b0324fe9acbe4d9dce3a33dc. Latest consensus state: current_term: 3 leader_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8521c085b0324fe9acbe4d9dce3a33dc" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45107 } } }
I20260808 02:27:52.456948 3629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:52.456960 3630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8521c085b0324fe9acbe4d9dce3a33dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:27:52.458107 3639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:27:52.476665 3639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:27:52.478595 3639 catalog_manager.cc:1259] Loaded cluster ID: 2ccbd78b7cc740a9a0af28d2cd4cba68
I20260808 02:27:52.478828 3639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 02:27:52.481410 3639 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f71fa9859ad google::LogMessage::Fail() at ??:0
I20260808 02:27:52.486336 3647 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f71fa98888a google::LogMessage::SendToLog() at ??:0
@ 0x7f71fa9854de google::LogMessage::Flush() at ??:0
@ 0x7f71fa986a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f72103bc512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f72103b9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7210441610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7210441439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f71fd5d1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f71fd5d491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f71fd5d4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f71fd5afeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f72008896db start_thread at ??:0
@ 0x7f71f981f61f clone at ??:0
W20260808 02:27:52.815207 3626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:52.815523 3626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:52.815745 3626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:52.823335 3626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:52.823493 3626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:52.823814 3626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:27:52.833913 3626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:45747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=39467
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:52.835271 3626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:52.836764 3626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:52.850415 3668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:52.850386 3669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:52.850348 3671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:52.851608 3626 server_base.cc:1034] running on GCE node
I20260808 02:27:52.852224 3626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:52.853186 3626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:52.854426 3626 hybrid_clock.cc:648] HybridClock initialized: now 1786156072854391 us; error 44 us; skew 500 ppm
I20260808 02:27:52.854815 3626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:52 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156072, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:52.858052 3626 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:27:52.859678 3626 webserver.cc:460] Webserver started at http://127.19.231.1:39467/ using document root <none> and password file <none>
I20260808 02:27:52.860368 3626 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:52.860529 3626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:52.865978 3626 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 02:27:52.868806 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:52.870201 3626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:27:52.870394 3626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "8610e243fe9e42509c218b61528f147d"
format_stamp: "Formatted at 2026-08-08 02:27:46 on dist-test-slave-pgkr"
server_key: "319cd5057bfd57335abe72e27cd14e8d"
server_key_iv: "24bd3b2dedd4ef31512943523d1ac5bf"
server_key_version: "encryptionkey@0"
I20260808 02:27:52.871513 3626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:52.890522 3626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:52.894999 3626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:52.895320 3626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:52.896445 3626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:52.898379 3685 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260808 02:27:52.903687 3685 ts_tablet_manager.cc:1852] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 02:27:52.904685 3685 ts_tablet_manager.cc:1905] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:27:52.912601 3685 ts_tablet_manager.cc:1918] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:27:52.912964 3685 ts_tablet_manager.cc:1939] T e95838de27b64686b23de16e2d866b83 P 8610e243fe9e42509c218b61528f147d: Deleting consensus metadata
I20260808 02:27:52.913504 3626 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 02:27:52.913650 3626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260808 02:27:52.913808 3626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:52.913906 3626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:52.949951 3626 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:45747
I20260808 02:27:52.950011 3801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:45747 every 8 connection(s)
I20260808 02:27:52.951387 3626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:27:52.956420 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:52.954496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:33651)
0808 02:27:52.955289 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:52.955529 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":297,"threads_started":1}
W20260808 02:27:52.957041 3802 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:45107 (0 consecutive failures): Network error: Failed to ping master at 127.19.231.62:45107: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
I20260808 02:27:52.957912 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:52.957617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:51633)
0808 02:27:52.957751 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:52.957815 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:27:52.958835 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:52.958434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:47685)
0808 02:27:52.958549 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:52.958666 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 02:27:52.959128 3802 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:27:52.959370 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 3626
I20260808 02:27:52.959962 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:40401
--local_ip_for_outbound_sockets=127.19.231.2
--tserver_master_addrs=127.19.231.62:45107
--webserver_port=39487
--webserver_interface=127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:27:53.329489 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:53.329820 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:53.330013 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:53.338174 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:53.338286 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:53.338442 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:27:53.348652 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:40401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=39487
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:53.350178 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:53.351655 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:53.365247 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:53.365304 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:53.365329 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:53.366391 3805 server_base.cc:1034] running on GCE node
I20260808 02:27:53.367126 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:53.368091 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:53.369340 3805 hybrid_clock.cc:648] HybridClock initialized: now 1786156073369281 us; error 48 us; skew 500 ppm
I20260808 02:27:53.369675 3805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:53 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156073, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:53.372890 3805 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:27:53.374495 3805 webserver.cc:460] Webserver started at http://127.19.231.2:39487/ using document root <none> and password file <none>
I20260808 02:27:53.375414 3805 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:53.375576 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:53.381193 3805 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 02:27:53.384123 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:53.385468 3805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:27:53.385690 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "d19e4a415bfe4852886ddfcefb4a8923"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "b87be77feb60d2cfe801c9277ba6faf7"
server_key_iv: "592ff9af1d4e175f204d74c95b7f0e09"
server_key_version: "encryptionkey@0"
I20260808 02:27:53.386785 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:53.405622 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:53.409875 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:53.410172 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:53.411265 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:53.412643 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:53.412786 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:53.412916 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:53.413007 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:53.451069 3805 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:40401
I20260808 02:27:53.451167 3941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:40401 every 8 connection(s)
I20260808 02:27:53.453154 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:27:53.458791 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:53.456810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:49565)
0808 02:27:53.457543 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:53.457763 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":284,"threads_started":1}
W20260808 02:27:53.459671 3942 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:45107 (0 consecutive failures): Network error: Failed to ping master at 127.19.231.62:45107: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
I20260808 02:27:53.460712 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:53.460321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:55883)
0808 02:27:53.460471 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:53.460563 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 02:27:53.460913 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 3805
I20260808 02:27:53.461560 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:44941
--local_ip_for_outbound_sockets=127.19.231.3
--tserver_master_addrs=127.19.231.62:45107
--webserver_port=34621
--webserver_interface=127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:27:53.463997 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:53.463497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:52821)
0808 02:27:53.463703 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:53.463821 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 02:27:53.464361 3942 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:27:53.822705 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:27:53.822999 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:27:53.823195 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:27:53.830431 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:27:53.830542 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:27:53.830785 3945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:27:53.840646 3945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:44941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=34621
--tserver_master_addrs=127.19.231.62:45107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:27:53.842051 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:27:53.843449 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:27:53.857110 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:53.857211 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:27:53.857394 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:27:53.858181 3945 server_base.cc:1034] running on GCE node
I20260808 02:27:53.858875 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:27:53.859856 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:27:53.861089 3945 hybrid_clock.cc:648] HybridClock initialized: now 1786156073861008 us; error 72 us; skew 500 ppm
I20260808 02:27:53.861421 3945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:27:53 dist-test-slave-pgkr krb5kdc[2394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156073, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:27:53.864638 3945 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:27:53.866052 3945 webserver.cc:460] Webserver started at http://127.19.231.3:34621/ using document root <none> and password file <none>
I20260808 02:27:53.866813 3945 fs_manager.cc:359] Metadata directory not provided
I20260808 02:27:53.866923 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:27:53.872705 3945 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 02:27:53.876024 3962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:53.877667 3945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:27:53.877847 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "dd714139ef754132a34d37e4fa4f9d3c"
format_stamp: "Formatted at 2026-08-08 02:27:47 on dist-test-slave-pgkr"
server_key: "db4c16a155a1caf2e2df0a3c88f6c22f"
server_key_iv: "ec0b37ed7e8e9794c8f1fc1c77e6b568"
server_key_version: "encryptionkey@0"
I20260808 02:27:53.878892 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:27:53.893020 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:27:53.897110 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:27:53.897379 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:27:53.898347 3945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:27:53.899899 3945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:27:53.900010 3945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:53.900134 3945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:27:53.900223 3945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:27:53.935290 3945 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:44941
I20260808 02:27:53.935568 4084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:44941 every 8 connection(s)
I20260808 02:27:53.936903 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:27:53.938428 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 3945
I20260808 02:27:53.942342 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:53.940437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:47085)
0808 02:27:53.941260 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:53.941479 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":293,"threads_started":1}
W20260808 02:27:53.943104 4085 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:45107 (0 consecutive failures): Network error: Failed to ping master at 127.19.231.62:45107: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
I20260808 02:27:53.943980 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:53.943631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:40975)
0808 02:27:53.943800 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:53.943875 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 02:27:53.944804 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:53.944503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:41371)
0808 02:27:53.944623 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:53.944698 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:27:53.945060 4085 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:27:53.960527 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:53.959667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:54479)
0808 02:27:53.960190 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:53.960287 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":289,"threads_started":1}
I20260808 02:27:54.466329 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:54.464972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:45045)
0808 02:27:54.465878 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:54.466031 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":531,"threads_started":1}
I20260808 02:27:54.946343 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:54.945568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:38721)
0808 02:27:54.946002 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:54.946113 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":216,"threads_started":1}
I20260808 02:27:54.962265 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:54.961506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:46631)
0808 02:27:54.961991 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:54.962089 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":271,"threads_started":1}
I20260808 02:27:55.468080 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:55.467293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:53129)
0808 02:27:55.467777 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:55.467861 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":262,"threads_started":1}
I20260808 02:27:55.948211 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:55.947263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:60563)
0808 02:27:55.947834 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:55.947954 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":298,"threads_started":1}
I20260808 02:27:55.963958 4097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:55.963189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:49389)
0808 02:27:55.963693 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:55.963790 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":235,"threads_started":1}
I20260808 02:27:56.469657 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:56.468916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:60011)
0808 02:27:56.469393 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:56.469474 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":277,"threads_started":1}
I20260808 02:27:56.949759 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:56.949055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:52243)
0808 02:27:56.949509 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:56.949590 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260808 02:27:56.966046 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:56.964962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:36579)
0808 02:27:56.965576 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:56.965803 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":347,"threads_started":1}
I20260808 02:27:57.471536 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:57.470610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:34591)
0808 02:27:57.471239 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:57.471362 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":255,"threads_started":1}
I20260808 02:27:57.951611 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:57.950780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:47437)
0808 02:27:57.951303 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:57.951385 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":311,"threads_started":1}
I20260808 02:27:57.967864 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:57.967005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:41773)
0808 02:27:57.967608 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:57.967692 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":327,"threads_started":1}
I20260808 02:27:58.473367 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:58.472613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:53007)
0808 02:27:58.473110 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:58.473196 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":247,"threads_started":1}
I20260808 02:27:58.953367 4105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:58.952533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:46239)
0808 02:27:58.953095 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:58.953206 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":243,"threads_started":1}
I20260808 02:27:58.969625 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:58.968771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:54871)
0808 02:27:58.969319 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:58.969419 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
I20260808 02:27:59.475262 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:59.474438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:32859)
0808 02:27:59.474998 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:59.475105 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":328,"threads_started":1}
I20260808 02:27:59.955083 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:59.954318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:34447)
0808 02:27:59.954832 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:59.954922 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":242,"threads_started":1}
I20260808 02:27:59.971575 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:27:59.970705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:38699)
0808 02:27:59.971274 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 02:27:59.971366 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":237,"threads_started":1}
I20260808 02:28:00.477288 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:00.476392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:49463)
0808 02:28:00.476981 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:00.477066 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":245,"threads_started":1}
I20260808 02:28:00.956845 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:00.956058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:34575)
0808 02:28:00.956553 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:00.956632 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":237,"threads_started":1}
I20260808 02:28:00.973243 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:00.972483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:42271)
0808 02:28:00.972975 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:00.973059 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":244,"threads_started":1}
I20260808 02:28:01.478994 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:01.478186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:39997)
0808 02:28:01.478729 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:01.478821 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
I20260808 02:28:01.958531 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:01.957776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:60669)
0808 02:28:01.958278 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:01.958366 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":235,"threads_started":1}
I20260808 02:28:01.975190 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:01.974275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:48911)
0808 02:28:01.974934 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:01.975027 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":278,"threads_started":1}
I20260808 02:28:02.480640 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:02.479892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:37467)
0808 02:28:02.480378 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:02.480464 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260808 02:28:02.960618 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:02.959766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:41635)
0808 02:28:02.960337 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:02.960431 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":290,"threads_started":1}
I20260808 02:28:02.976917 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:02.976099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:37273)
0808 02:28:02.976640 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:02.976728 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":306,"threads_started":1}
I20260808 02:28:03.482662 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:03.481702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:43385)
0808 02:28:03.482320 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:03.482407 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":338,"threads_started":1}
I20260808 02:28:03.962285 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:03.961526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:54077)
0808 02:28:03.962023 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:03.962127 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":303,"threads_started":1}
I20260808 02:28:03.978821 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:03.977896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:38329)
0808 02:28:03.978474 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:03.978558 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":351,"threads_started":1}
I20260808 02:28:04.484351 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:04.483584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:39427)
0808 02:28:04.484067 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:04.484163 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
I20260808 02:28:04.964013 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:04.963301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:54405)
0808 02:28:04.963735 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:04.963827 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":214,"threads_started":1}
I20260808 02:28:04.980368 4125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:04.979676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:44555)
0808 02:28:04.980118 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:04.980199 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
I20260808 02:28:05.486116 4126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:05.485305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:47973)
0808 02:28:05.485828 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:05.485929 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":248,"threads_started":1}
I20260808 02:28:05.965773 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:05.964993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:47035)
0808 02:28:05.965500 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:05.965608 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
I20260808 02:28:05.982170 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:05.981338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:37841)
0808 02:28:05.981809 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:05.981932 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260808 02:28:06.487959 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:06.487086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:58985)
0808 02:28:06.487647 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:06.487751 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
I20260808 02:28:06.967855 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:06.966837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:57149)
0808 02:28:06.967400 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:06.967576 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260808 02:28:06.983963 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:06.983074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:49117)
0808 02:28:06.983652 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:06.983750 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":316,"threads_started":1}
I20260808 02:28:07.489974 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:07.488939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:47437)
0808 02:28:07.489595 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:07.489730 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":321,"threads_started":1}
I20260808 02:28:07.969677 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:07.968886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:40065)
0808 02:28:07.969348 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:07.969441 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":240,"threads_started":1}
I20260808 02:28:07.985966 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:07.985197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.1:54663)
0808 02:28:07.985695 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:07.985787 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":219,"threads_started":1}
I20260808 02:28:08.491988 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:08.491236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.2:52247)
0808 02:28:08.491700 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:08.491793 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260808 02:28:08.942018 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 3626
I20260808 02:28:08.968132 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:08.943377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.19.231.1:45747)
0808 02:28:08.944294 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:08.944310 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:08.944351 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:28:08.946258 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:08.946270 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:08.946807 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:08.947230 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:08.948878 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:08.950253 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:08.951825 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:08.952622 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:08.952863 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:08.961229 (+ 8366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:08.961238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:08.961255 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:08.961317 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:08.964391 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:08.965436 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:08.965443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:08.965450 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:08.965630 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:08.966026 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:08.966029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:08.966031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:08.966592 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:08.966983 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:08.967464 (+ 481us) server_negotiation.cc:299] Negotiation successful
0808 02:28:08.967823 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":494,"threads_started":1}
I20260808 02:28:08.969646 3797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:08.975575 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:08.970907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45107 (local address 127.19.231.3:39989)
0808 02:28:08.975121 (+ 4214us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:08.975273 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4037,"thread_start_us":3890,"threads_started":1}
I20260808 02:28:09.061443 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 3626
I20260808 02:28:09.086293 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 3805
I20260808 02:28:09.104948 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:09.087374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.19.231.2:40401)
0808 02:28:09.087918 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:09.087933 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:09.087972 (+ 39us) server_negotiation.cc:407] Connection header received
0808 02:28:09.089655 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:09.089665 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:09.090208 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:09.090543 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:09.092066 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:09.093499 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:09.094816 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:09.095557 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:09.095714 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:09.098319 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:09.098327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:09.098343 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:09.098385 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:09.101633 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:09.102261 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:09.102268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:09.102274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:09.102464 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:09.102917 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:09.102921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:09.102923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:09.103473 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:09.103759 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:09.104153 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 02:28:09.104601 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260808 02:28:09.106437 3937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:09.192096 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 3805
I20260808 02:28:09.215077 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 3945
I20260808 02:28:09.234100 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:09.216286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.19.231.3:44941)
0808 02:28:09.216949 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:09.216963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:09.217007 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:28:09.218802 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:09.218812 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:09.219271 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:09.219568 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:09.220918 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:09.222261 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:09.223746 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:09.224503 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:09.224699 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:09.227465 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:09.227474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:09.227490 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:09.227527 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:09.230933 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:09.231547 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:09.231557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:09.231566 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:09.231746 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:09.232099 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:09.232103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:09.232105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:09.232697 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:09.232953 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:09.233261 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 02:28:09.233772 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":311,"threads_started":1}
I20260808 02:28:09.235671 4080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:09.324882 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 3945
2026-08-08T02:28:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25215 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:28:09 dist-test-slave-pgkr krb5kdc[4146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:28:09 dist-test-slave-pgkr krb5kdc[4146](info): set up 2 sockets
Aug 08 02:28:09 dist-test-slave-pgkr krb5kdc[4146](info): commencing operation
krb5kdc: starting...
W20260808 02:28:10.501097 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:28:10 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:28:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:28:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:28:10.827761 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43731
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:28:11.179903 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:11.180197 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:11.180370 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:11.187747 4159 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:28:11.187867 4159 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:28:11.187958 4159 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:28:11.188112 4159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:28:11.188206 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:11.188323 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:11.188397 4159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:28:11.188481 4159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:28:11.198439 4159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:11.199851 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:11.201244 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:11.213753 4164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:11.213721 4165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:11.213721 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:11.214643 4159 server_base.cc:1034] running on GCE node
I20260808 02:28:11.215476 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:11.216943 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:11.218189 4159 hybrid_clock.cc:648] HybridClock initialized: now 1786156091218118 us; error 42 us; skew 500 ppm
I20260808 02:28:11.218526 4159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:11 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156091, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:11.222160 4159 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:28:11.224012 4159 webserver.cc:460] Webserver started at http://127.19.231.62:40031/ using document root <none> and password file <none>
I20260808 02:28:11.224846 4159 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:11.224994 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:11.225373 4159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:28:11.228868 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "be34fb3fd0ec42bcaf21c5cceac254e0"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "2b117df7d8144eb75f85ed4ec440dbf5"
server_key_iv: "db4214085df149ec067340e4d7e3e3e7"
server_key_version: "encryptionkey@0"
I20260808 02:28:11.229753 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "be34fb3fd0ec42bcaf21c5cceac254e0"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "2b117df7d8144eb75f85ed4ec440dbf5"
server_key_iv: "db4214085df149ec067340e4d7e3e3e7"
server_key_version: "encryptionkey@0"
I20260808 02:28:11.235472 4159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260808 02:28:11.239272 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:11.240643 4159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:28:11.240849 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "be34fb3fd0ec42bcaf21c5cceac254e0"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "2b117df7d8144eb75f85ed4ec440dbf5"
server_key_iv: "db4214085df149ec067340e4d7e3e3e7"
server_key_version: "encryptionkey@0"
I20260808 02:28:11.241086 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:11.264962 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:11.269985 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:11.270298 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:11.291457 4159 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43731
I20260808 02:28:11.291507 4230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43731 every 8 connection(s)
I20260808 02:28:11.293012 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:28:11.296906 4231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:28:11.299382 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 4159
I20260808 02:28:11.299620 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:28:11.300485 20380 external_mini_cluster.cc:1428] Setting key 013b57ddf23e649d75afc764ee6af1df
I20260808 02:28:11.310072 4231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Bootstrap starting.
I20260808 02:28:11.314035 4231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Neither blocks nor log segments found. Creating new log.
I20260808 02:28:11.315505 4231 log.cc:824] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:28:11 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:28:11.318799 4231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: No bootstrap required, opened a new log
I20260808 02:28:11.326774 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:11.302742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.19.231.62:43731)
0808 02:28:11.303605 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:11.303647 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:11.303694 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:28:11.305421 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:11.305445 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:11.305854 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:11.306209 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:11.308141 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:11.309512 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:11.311450 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:11.312180 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:11.312344 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:11.317764 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:11.317771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:11.317785 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:11.317819 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:11.321877 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:11.322656 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:11.322668 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:11.322679 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:11.322889 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:11.323304 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:11.323309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:11.323312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:11.323992 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:11.324307 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:11.324680 (+ 373us) server_negotiation.cc:299] Negotiation successful
0808 02:28:11.325116 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":405,"threads_started":1}
I20260808 02:28:11.329775 4231 raft_consensus.cc:348] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:11.330202 4231 raft_consensus.cc:374] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:28:11.330281 4231 raft_consensus.cc:729] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be34fb3fd0ec42bcaf21c5cceac254e0, State: Initialized, Role: FOLLOWER
I20260808 02:28:11.330966 4231 consensus_queue.cc:260] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:11.331187 4231 raft_consensus.cc:388] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:28:11.331285 4231 raft_consensus.cc:482] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:28:11.331405 4231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:28:11.333541 4231 raft_consensus.cc:504] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:11.334111 4231 leader_election.cc:302] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be34fb3fd0ec42bcaf21c5cceac254e0; no voters:
I20260808 02:28:11.334810 4231 leader_election.cc:288] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:28:11.334944 4236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:28:11.335793 4236 raft_consensus.cc:686] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 LEADER]: Becoming Leader. State: Replica: be34fb3fd0ec42bcaf21c5cceac254e0, State: Running, Role: LEADER
I20260808 02:28:11.336362 4236 consensus_queue.cc:237] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:11.337029 4231 sys_catalog.cc:564] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:28:11.339567 4238 sys_catalog.cc:455] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be34fb3fd0ec42bcaf21c5cceac254e0. Latest consensus state: current_term: 1 leader_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } } }
I20260808 02:28:11.339968 4238 sys_catalog.cc:458] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:28:11.340442 4237 sys_catalog.cc:455] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } } }
I20260808 02:28:11.340703 4237 sys_catalog.cc:458] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:28:11.344821 4243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:28:11.351090 4243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:28:11.360834 4243 catalog_manager.cc:1347] Generated new cluster ID: 0f3947d710bb454bb5b8f2c3a1ae576c
I20260808 02:28:11.360963 4243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:28:11.396376 4243 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:28:11.397730 4243 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:28:11.404389 4243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Generated new TSK 0
I20260808 02:28:11.405220 4243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:28:11.488229 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43731
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:28:11.853322 4259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:11.853696 4259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:11.853875 4259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:11.862203 4259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:11.862376 4259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:11.862572 4259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:28:11.873728 4259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:11.875176 4259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:11.876642 4259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:11.891253 4264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:11.891243 4267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:11.891196 4265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:11.892236 4259 server_base.cc:1034] running on GCE node
I20260808 02:28:11.892920 4259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:11.893963 4259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:11.895210 4259 hybrid_clock.cc:648] HybridClock initialized: now 1786156091895154 us; error 48 us; skew 500 ppm
I20260808 02:28:11.895553 4259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:11 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156091, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:11.899109 4259 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:28:11.900695 4259 webserver.cc:460] Webserver started at http://127.19.231.1:46635/ using document root <none> and password file <none>
I20260808 02:28:11.901419 4259 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:11.901576 4259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:11.901929 4259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:28:11.905364 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "b21c8493940d43baa9c6ff45b67b1556"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "4f976887f0398064fc8ea86a3e43a693"
server_key_iv: "eacf72fea3ff6f144a9fa907ed026941"
server_key_version: "encryptionkey@0"
I20260808 02:28:11.906174 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "b21c8493940d43baa9c6ff45b67b1556"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "4f976887f0398064fc8ea86a3e43a693"
server_key_iv: "eacf72fea3ff6f144a9fa907ed026941"
server_key_version: "encryptionkey@0"
I20260808 02:28:11.911718 4259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:28:11.915432 4274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:11.916828 4259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:28:11.917024 4259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "b21c8493940d43baa9c6ff45b67b1556"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "4f976887f0398064fc8ea86a3e43a693"
server_key_iv: "eacf72fea3ff6f144a9fa907ed026941"
server_key_version: "encryptionkey@0"
I20260808 02:28:11.917232 4259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:11.937450 4259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:11.942009 4259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:11.942314 4259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:11.943320 4259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:11.944738 4259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:11.944890 4259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 02:28:11.945070 4259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:11.945148 4259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:11.979868 4259 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:43323
I20260808 02:28:11.979915 4397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:43323 every 8 connection(s)
I20260808 02:28:11.981215 4259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:28:11.981884 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 4259
I20260808 02:28:11.982126 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:28:11.982982 20380 external_mini_cluster.cc:1428] Setting key 65bd42adda13aa4ed6a4824014698cb9
Aug 08 02:28:11 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156091, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:28:12.003917 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:11.984157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:38749 (local address 127.19.231.62:43731)
0808 02:28:11.984608 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:11.984612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:11.987252 (+ 2640us) server_negotiation.cc:407] Connection header received
0808 02:28:11.989317 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:11.989322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:11.989414 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:11.989551 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:11.991181 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:11.991898 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:11.993470 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:11.993870 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:11.993998 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:11.997892 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:11.997897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:11.997900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:11.997934 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:12.000540 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:12.001498 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:12.001502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:12.001504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:12.001619 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:12.002102 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:12.002105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:12.002107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:12.002364 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:12.002473 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:12.003467 (+ 994us) server_negotiation.cc:299] Negotiation successful
0808 02:28:12.003646 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":253,"threads_started":1}
I20260808 02:28:12.005614 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:11.984834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:38749)
0808 02:28:11.987106 (+ 2272us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:11.987145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:11.989039 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:11.989809 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:11.989818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:11.990342 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:28:11.991009 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:11.991028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:11.992070 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:11.992075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:11.993314 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:11.993326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:11.994114 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:11.994122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:11.994361 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:11.995104 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:28:11.995132 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:28:11.997662 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:28:12.000692 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:12.000703 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:12.000722 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:12.001375 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:12.001732 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:12.001740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:12.001746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:12.001993 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:12.002492 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:28:12.002503 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:28:12.003055 (+ 552us) client_negotiation.cc:769] Sending connection context
0808 02:28:12.003366 (+ 311us) client_negotiation.cc:240] Negotiation successful
0808 02:28:12.003979 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2016,"thread_start_us":1829,"threads_started":1}
I20260808 02:28:12.007382 4399 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43731
I20260808 02:28:12.007717 4399 heartbeater.cc:464] Registering TS with master...
I20260808 02:28:12.008514 4399 heartbeater.cc:511] Master 127.19.231.62:43731 requested a full tablet report, sending...
I20260808 02:28:12.010962 4194 ts_manager.cc:194] Registered new tserver with Master: b21c8493940d43baa9c6ff45b67b1556 (127.19.231.1:43323)
I20260808 02:28:12.012428 4194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:38749
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:28:12.053773 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43731
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:28:12.409320 4405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:12.409680 4405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:12.409864 4405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:12.417222 4405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:12.417383 4405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:12.417554 4405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:28:12.427802 4405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:12.429132 4405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:12.430526 4405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:12.444547 4410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:12.444631 4414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:12.444691 4411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:12.445899 4405 server_base.cc:1034] running on GCE node
I20260808 02:28:12.446741 4405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:12.447710 4405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:12.448961 4405 hybrid_clock.cc:648] HybridClock initialized: now 1786156092448914 us; error 33 us; skew 500 ppm
I20260808 02:28:12.449316 4405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:12 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156092, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:12.452853 4405 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:28:12.454440 4405 webserver.cc:460] Webserver started at http://127.19.231.2:33003/ using document root <none> and password file <none>
I20260808 02:28:12.455268 4405 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:12.455443 4405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:12.455806 4405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:28:12.459082 4405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "5e36bc5416f04b90b10c1bf4d0da03aa"
format_stamp: "Formatted at 2026-08-08 02:28:12 on dist-test-slave-pgkr"
server_key: "db47ceb460225e22bf5539d1fd68643c"
server_key_iv: "5dc6f966f94dea88977a75eaf30c3d13"
server_key_version: "encryptionkey@0"
I20260808 02:28:12.459915 4405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "5e36bc5416f04b90b10c1bf4d0da03aa"
format_stamp: "Formatted at 2026-08-08 02:28:12 on dist-test-slave-pgkr"
server_key: "db47ceb460225e22bf5539d1fd68643c"
server_key_iv: "5dc6f966f94dea88977a75eaf30c3d13"
server_key_version: "encryptionkey@0"
I20260808 02:28:12.465590 4405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:28:12.469521 4422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:12.471028 4405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:28:12.471222 4405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "5e36bc5416f04b90b10c1bf4d0da03aa"
format_stamp: "Formatted at 2026-08-08 02:28:12 on dist-test-slave-pgkr"
server_key: "db47ceb460225e22bf5539d1fd68643c"
server_key_iv: "5dc6f966f94dea88977a75eaf30c3d13"
server_key_version: "encryptionkey@0"
I20260808 02:28:12.471438 4405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:12.498112 4405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:12.503242 4405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:12.503580 4405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:12.504731 4405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:12.506212 4405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:12.506327 4405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:12.506505 4405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:12.506589 4405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:12.541999 4405 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:35445
I20260808 02:28:12.542094 4546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:35445 every 8 connection(s)
I20260808 02:28:12.543629 4405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:28:12.546155 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 4405
I20260808 02:28:12.546728 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:28:12.547626 20380 external_mini_cluster.cc:1428] Setting key f16de49e4a087408957f13fbd7424e16
Aug 08 02:28:12 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156092, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:28:12.580293 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:12.547462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:40119 (local address 127.19.231.62:43731)
0808 02:28:12.563725 (+ 16263us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:12.563731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:12.563752 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:28:12.563813 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:12.563818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:12.563898 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:12.564021 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:12.565930 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:12.566749 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:12.568307 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:12.568864 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:12.568995 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:12.572735 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:12.572744 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:12.572752 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:12.572798 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:12.576725 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:12.577525 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:12.577533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:12.577539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:12.577711 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:12.578161 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:12.578169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:12.578175 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:12.578557 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:12.578763 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:12.579494 (+ 731us) server_negotiation.cc:299] Negotiation successful
0808 02:28:12.579757 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16120,"thread_start_us":16657,"threads_started":1}
I20260808 02:28:12.581064 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:12.548327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:40119)
0808 02:28:12.559921 (+ 11594us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:12.559984 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:12.563387 (+ 3403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:12.564320 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:12.564338 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:12.565163 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:28:12.565746 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:12.565764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:12.566894 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:12.566900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:12.568149 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:12.568171 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:12.569112 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:12.569118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:12.569324 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:12.570093 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:28:12.570116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:28:12.572462 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:28:12.576889 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:12.576896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:12.576910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:12.577391 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:12.577826 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:12.577829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:12.577832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:12.578040 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:12.578749 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:28:12.578755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:28:12.579106 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 02:28:12.579367 (+ 261us) client_negotiation.cc:240] Negotiation successful
0808 02:28:12.579793 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11229,"thread_start_us":10972,"threads_started":1}
I20260808 02:28:12.582726 4547 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43731
I20260808 02:28:12.583098 4547 heartbeater.cc:464] Registering TS with master...
I20260808 02:28:12.584082 4547 heartbeater.cc:511] Master 127.19.231.62:43731 requested a full tablet report, sending...
I20260808 02:28:12.585881 4194 ts_manager.cc:194] Registered new tserver with Master: 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445)
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
I20260808 02:28:12.586874 4194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:40119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:28:12.626441 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43731
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:28:12.975126 4554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:12.975400 4554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:12.975497 4554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:12.982836 4554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:12.982940 4554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:12.983120 4554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:28:12.993113 4554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:12.994441 4554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:12.995847 4554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:13.010201 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:13.010175 4559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:13.010119 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:13.011066 4554 server_base.cc:1034] running on GCE node
I20260808 02:28:13.011861 4554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:13.012877 4554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:13.014120 4554 hybrid_clock.cc:648] HybridClock initialized: now 1786156093014041 us; error 47 us; skew 500 ppm
I20260808 02:28:13.014488 4554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:28:13.015683 4399 heartbeater.cc:503] Master 127.19.231.62:43731 was elected leader, sending a full tablet report...
Aug 08 02:28:13 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:13.018770 4554 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:28:13.020534 4554 webserver.cc:460] Webserver started at http://127.19.231.3:38611/ using document root <none> and password file <none>
I20260808 02:28:13.021358 4554 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:13.021603 4554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:13.021955 4554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:28:13.025308 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "c39eebd671cc45559fa6705a71c27222"
format_stamp: "Formatted at 2026-08-08 02:28:13 on dist-test-slave-pgkr"
server_key: "4452f183100bbffd3528d8dfc218066c"
server_key_iv: "6cea29e13644d8f9aa47634e3fb8b7df"
server_key_version: "encryptionkey@0"
I20260808 02:28:13.026186 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "c39eebd671cc45559fa6705a71c27222"
format_stamp: "Formatted at 2026-08-08 02:28:13 on dist-test-slave-pgkr"
server_key: "4452f183100bbffd3528d8dfc218066c"
server_key_iv: "6cea29e13644d8f9aa47634e3fb8b7df"
server_key_version: "encryptionkey@0"
I20260808 02:28:13.031767 4554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 02:28:13.035291 4571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:13.036612 4554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:28:13.036829 4554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "c39eebd671cc45559fa6705a71c27222"
format_stamp: "Formatted at 2026-08-08 02:28:13 on dist-test-slave-pgkr"
server_key: "4452f183100bbffd3528d8dfc218066c"
server_key_iv: "6cea29e13644d8f9aa47634e3fb8b7df"
server_key_version: "encryptionkey@0"
I20260808 02:28:13.037024 4554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:13.057113 4554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:13.061324 4554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:13.061638 4554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:13.062661 4554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:13.064121 4554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:13.064240 4554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:13.064388 4554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:13.064467 4554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:13.099493 4554 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:42377
I20260808 02:28:13.099587 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:42377 every 8 connection(s)
I20260808 02:28:13.100816 4554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:28:13.108816 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 4554
I20260808 02:28:13.109014 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:28:13.109472 20380 external_mini_cluster.cc:1428] Setting key 6e78dba93a2195d71f02f2f5e8322c46
Aug 08 02:28:13 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:28:13.122154 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.103722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:43903 (local address 127.19.231.62:43731)
0808 02:28:13.104239 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.104246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.105248 (+ 1002us) server_negotiation.cc:407] Connection header received
0808 02:28:13.107260 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.107265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.107345 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.107443 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:13.109111 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.110088 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.112275 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.112935 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.113091 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.116427 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:13.116432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:13.116436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:13.116468 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:13.119244 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.119930 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.119935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.119937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.120097 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.120478 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.120481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.120483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.120839 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:13.120973 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.121625 (+ 652us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.121857 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260808 02:28:13.123139 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.104329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:43903)
0808 02:28:13.105082 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:13.105127 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:13.106991 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:13.107712 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:13.107721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:13.108271 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:28:13.108856 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.108875 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.110279 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.110286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.112108 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.112121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.113200 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.113210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.113435 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.114039 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:28:13.114065 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:28:13.116237 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:28:13.119374 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:13.119380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:13.119392 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:13.119800 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:13.120195 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:13.120198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:13.120201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:13.120371 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:13.120955 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:28:13.120961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:28:13.121267 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 02:28:13.121481 (+ 214us) client_negotiation.cc:240] Negotiation successful
0808 02:28:13.121936 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":287,"threads_started":1}
I20260808 02:28:13.124547 4693 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43731
I20260808 02:28:13.124874 4693 heartbeater.cc:464] Registering TS with master...
I20260808 02:28:13.125574 4693 heartbeater.cc:511] Master 127.19.231.62:43731 requested a full tablet report, sending...
I20260808 02:28:13.127025 4194 ts_manager.cc:194] Registered new tserver with Master: c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377)
I20260808 02:28:13.127835 4194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:43903
I20260808 02:28:13.138011 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:28:13.155699 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.142556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.19.231.62:43731)
0808 02:28:13.142748 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.142753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.143263 (+ 510us) server_negotiation.cc:407] Connection header received
0808 02:28:13.143414 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.143417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.143480 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.143561 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:13.145033 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.146140 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.147642 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.148049 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.148157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.150548 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:13.150553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:13.150556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:13.150587 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:13.153345 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.153944 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.153948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.153950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.154078 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.154396 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.154399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.154401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.154790 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:13.154973 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.155253 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.155420 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:28:13.159761 4194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:28:13.166867 4194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:28:13.190928 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.179783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.19.231.2:35445)
0808 02:28:13.180838 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.180847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.181101 (+ 254us) server_negotiation.cc:407] Connection header received
0808 02:28:13.181372 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.181381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.181685 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.181935 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:13.183002 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.184961 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.187942 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.188976 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.189195 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.189456 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.190357 (+ 901us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.190547 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":445,"threads_started":1}
I20260808 02:28:13.192150 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.179712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:35445 (local address 127.0.0.1:59924)
0808 02:28:13.180936 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:13.180980 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:13.181157 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:13.181974 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:13.181986 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:13.182026 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:13.182705 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.182728 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.185138 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.185147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.187767 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.187781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.189638 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.189645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.190056 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.190131 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 02:28:13.190279 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 02:28:13.190488 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":53,"thread_start_us":860,"threads_started":1}
I20260808 02:28:13.193265 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.181396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:42377 (local address 127.0.0.1:33328)
0808 02:28:13.182694 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:13.182712 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:13.183964 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:13.185598 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:13.185602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:13.185622 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:13.186008 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.186018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.188058 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.188064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.189993 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.190005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.191952 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.191959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.192174 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.192211 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:28:13.192717 (+ 506us) client_negotiation.cc:240] Negotiation successful
0808 02:28:13.192847 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":129,"thread_start_us":968,"threads_started":1}
I20260808 02:28:13.193312 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.181779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.19.231.3:42377)
0808 02:28:13.184628 (+ 2849us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.184641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.184685 (+ 44us) server_negotiation.cc:407] Connection header received
0808 02:28:13.184778 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.184785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.185066 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.185289 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:13.186221 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.187866 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.190149 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.191734 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.191889 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.192132 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.192772 (+ 640us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.192957 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2642,"thread_start_us":2462,"threads_started":1}
I20260808 02:28:13.195890 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.182971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.19.231.1:43323)
0808 02:28:13.185730 (+ 2759us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.185741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.185773 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:28:13.185877 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.185888 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.186128 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.186350 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:13.187373 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.188960 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.191964 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.194787 (+ 2823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.194936 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.195176 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.195381 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.195571 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2573,"thread_start_us":2389,"threads_started":1}
I20260808 02:28:13.196389 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.183428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:43323 (local address 127.0.0.1:48408)
0808 02:28:13.184940 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:13.184960 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:13.185101 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:13.186697 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:13.186709 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:13.186740 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:13.187172 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.187185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.189149 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.189154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.191749 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.191765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.194991 (+ 3226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.194998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.195177 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.195216 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:28:13.195319 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:28:13.195472 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":101,"thread_start_us":1242,"threads_started":1}
I20260808 02:28:13.196833 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5052d0319c4827a5d2d27e4be5dc34 (DEFAULT_TABLE table=test-table [id=755507a3df7b4abea1ec4449346b8627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:28:13.197247 4623 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5052d0319c4827a5d2d27e4be5dc34 (DEFAULT_TABLE table=test-table [id=755507a3df7b4abea1ec4449346b8627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:28:13.198604 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5052d0319c4827a5d2d27e4be5dc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:28:13.198765 4623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5052d0319c4827a5d2d27e4be5dc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:28:13.199527 4327 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5052d0319c4827a5d2d27e4be5dc34 (DEFAULT_TABLE table=test-table [id=755507a3df7b4abea1ec4449346b8627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:28:13.200680 4327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5052d0319c4827a5d2d27e4be5dc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:28:13.213749 4714 tablet_bootstrap.cc:492] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Bootstrap starting.
I20260808 02:28:13.215677 4716 tablet_bootstrap.cc:492] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Bootstrap starting.
I20260808 02:28:13.216456 4717 tablet_bootstrap.cc:492] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Bootstrap starting.
I20260808 02:28:13.217351 4714 tablet_bootstrap.cc:654] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Neither blocks nor log segments found. Creating new log.
I20260808 02:28:13.218791 4714 log.cc:824] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:13.220144 4717 tablet_bootstrap.cc:654] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Neither blocks nor log segments found. Creating new log.
I20260808 02:28:13.221479 4717 log.cc:824] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:13.222499 4716 tablet_bootstrap.cc:654] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Neither blocks nor log segments found. Creating new log.
I20260808 02:28:13.223241 4714 tablet_bootstrap.cc:492] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: No bootstrap required, opened a new log
I20260808 02:28:13.223594 4714 ts_tablet_manager.cc:1397] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260808 02:28:13.223743 4716 log.cc:824] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:13.227011 4717 tablet_bootstrap.cc:492] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: No bootstrap required, opened a new log
I20260808 02:28:13.227098 4716 tablet_bootstrap.cc:492] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: No bootstrap required, opened a new log
I20260808 02:28:13.227329 4717 ts_tablet_manager.cc:1397] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260808 02:28:13.227373 4716 ts_tablet_manager.cc:1397] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260808 02:28:13.234537 4714 raft_consensus.cc:348] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.235044 4714 raft_consensus.cc:374] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:28:13.235142 4714 raft_consensus.cc:729] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21c8493940d43baa9c6ff45b67b1556, State: Initialized, Role: FOLLOWER
I20260808 02:28:13.235833 4714 consensus_queue.cc:260] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.236066 4717 raft_consensus.cc:348] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.236611 4717 raft_consensus.cc:374] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:28:13.236735 4717 raft_consensus.cc:729] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36bc5416f04b90b10c1bf4d0da03aa, State: Initialized, Role: FOLLOWER
I20260808 02:28:13.237461 4717 consensus_queue.cc:260] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.237993 4714 ts_tablet_manager.cc:1428] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260808 02:28:13.238138 4716 raft_consensus.cc:348] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.238725 4716 raft_consensus.cc:374] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:28:13.239087 4547 heartbeater.cc:503] Master 127.19.231.62:43731 was elected leader, sending a full tablet report...
I20260808 02:28:13.238831 4716 raft_consensus.cc:729] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39eebd671cc45559fa6705a71c27222, State: Initialized, Role: FOLLOWER
I20260808 02:28:13.239804 4717 ts_tablet_manager.cc:1428] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260808 02:28:13.240211 4716 consensus_queue.cc:260] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.243841 4693 heartbeater.cc:503] Master 127.19.231.62:43731 was elected leader, sending a full tablet report...
I20260808 02:28:13.244040 4716 ts_tablet_manager.cc:1428] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260808 02:28:13.315686 4548 tablet.cc:2369] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:28:13.352600 4694 tablet.cc:2369] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:28:13.440858 4722 raft_consensus.cc:482] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:28:13.441284 4722 raft_consensus.cc:504] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.443396 4722 leader_election.cc:288] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445), c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377)
I20260808 02:28:13.452392 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.443761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:58529 (local address 127.19.231.2:35445)
0808 02:28:13.443947 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.443952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.444571 (+ 619us) server_negotiation.cc:407] Connection header received
0808 02:28:13.444880 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.444889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.444989 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.445103 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:13.445918 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.446954 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.449611 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.450861 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.451041 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.451134 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.451982 (+ 848us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.452110 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:28:13.452392 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.444851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:45619 (local address 127.19.231.3:42377)
0808 02:28:13.445072 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.445079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.445408 (+ 329us) server_negotiation.cc:407] Connection header received
0808 02:28:13.445642 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.445647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.445731 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.445824 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:13.446455 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.447465 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.449789 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.451026 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.451168 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.451238 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.451982 (+ 744us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.452108 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:28:13.452704 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.443750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:35445 (local address 127.19.231.1:58529)
0808 02:28:13.444432 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:13.444453 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:13.444627 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:13.445337 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:13.445340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:13.445363 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:13.445648 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.445656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.447096 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.447103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.449446 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.449460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.451171 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.451178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.451662 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.451826 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 02:28:13.451944 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:28:13.452148 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":305,"thread_start_us":408,"threads_started":1}
I20260808 02:28:13.452755 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.444625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:42377 (local address 127.19.231.1:45619)
0808 02:28:13.445315 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:13.445329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:13.445481 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:13.445924 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:13.445927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:13.445947 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:13.446294 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.446306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.447598 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.447604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.449660 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:13.449669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.451232 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.451238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:13.451664 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.451826 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 02:28:13.451944 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 02:28:13.452148 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":189,"thread_start_us":304,"threads_started":1}
I20260808 02:28:13.453842 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5052d0319c4827a5d2d27e4be5dc34" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" is_pre_election: true
I20260808 02:28:13.454032 4643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5052d0319c4827a5d2d27e4be5dc34" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39eebd671cc45559fa6705a71c27222" is_pre_election: true
I20260808 02:28:13.454604 4643 raft_consensus.cc:2454] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 0.
I20260808 02:28:13.454586 4499 raft_consensus.cc:2454] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 0.
I20260808 02:28:13.455374 4277 leader_election.cc:302] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b21c8493940d43baa9c6ff45b67b1556, c39eebd671cc45559fa6705a71c27222; no voters:
I20260808 02:28:13.455770 4722 raft_consensus.cc:2781] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:28:13.455952 4722 raft_consensus.cc:482] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:28:13.456086 4722 raft_consensus.cc:3037] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:28:13.458470 4722 raft_consensus.cc:504] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.459276 4722 leader_election.cc:288] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 election: Requested vote from peers 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445), c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377)
I20260808 02:28:13.459749 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5052d0319c4827a5d2d27e4be5dc34" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa"
I20260808 02:28:13.459831 4643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5052d0319c4827a5d2d27e4be5dc34" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39eebd671cc45559fa6705a71c27222"
I20260808 02:28:13.459993 4499 raft_consensus.cc:3037] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:28:13.460035 4643 raft_consensus.cc:3037] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:28:13.463548 4499 raft_consensus.cc:2454] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 1.
I20260808 02:28:13.463549 4643 raft_consensus.cc:2454] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 1.
I20260808 02:28:13.463971 4277 leader_election.cc:302] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e36bc5416f04b90b10c1bf4d0da03aa, b21c8493940d43baa9c6ff45b67b1556; no voters:
I20260808 02:28:13.464315 4722 raft_consensus.cc:2781] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:28:13.464907 4722 raft_consensus.cc:686] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 1 LEADER]: Becoming Leader. State: Replica: b21c8493940d43baa9c6ff45b67b1556, State: Running, Role: LEADER
I20260808 02:28:13.465605 4722 consensus_queue.cc:237] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } }
I20260808 02:28:13.470580 4193 catalog_manager.cc:5495] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 reported cstate change: term changed from 0 to 1, leader changed from <none> to b21c8493940d43baa9c6ff45b67b1556 (127.19.231.1). New cstate: current_term: 1 leader_uuid: "b21c8493940d43baa9c6ff45b67b1556" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } health_report { overall_health: HEALTHY } } }
W20260808 02:28:13.484021 4400 tablet.cc:2369] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:28:13.490535 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.485000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.19.231.1:43323)
0808 02:28:13.485221 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.485226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.485304 (+ 78us) server_negotiation.cc:407] Connection header received
0808 02:28:13.485533 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.485537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.485633 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.485759 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:28:13.486108 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.486925 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.488545 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.488986 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.489114 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.489564 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:28:13.490065 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:28:13.490177 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.490275 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.490343 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 02:28:13.498981 4643 raft_consensus.cc:1261] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Refusing update from remote peer b21c8493940d43baa9c6ff45b67b1556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:28:13.498981 4499 raft_consensus.cc:1261] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Refusing update from remote peer b21c8493940d43baa9c6ff45b67b1556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:28:13.499962 4722 consensus_queue.cc:1035] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:28:13.500285 4727 consensus_queue.cc:1035] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:28:13.512805 4734 mvcc.cc:205] Tried to move back new op lower bound from 7316095358960934912 to 7316095358839853056. Current Snapshot: MvccSnapshot[applied={T|T < 7316095358960934912}]
I20260808 02:28:13.516861 4736 mvcc.cc:205] Tried to move back new op lower bound from 7316095358960934912 to 7316095358839853056. Current Snapshot: MvccSnapshot[applied={T|T < 7316095358960934912}]
I20260808 02:28:13.525708 4194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:28:13.525975 4194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:28:13.529863 4194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Sending DeleteTablet for 3 replicas of tablet 7e5052d0319c4827a5d2d27e4be5dc34
I20260808 02:28:13.531195 4477 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5052d0319c4827a5d2d27e4be5dc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:28:13 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:59924
I20260808 02:28:13.531200 4623 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5052d0319c4827a5d2d27e4be5dc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:28:13 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:33328
I20260808 02:28:13.531455 4327 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5052d0319c4827a5d2d27e4be5dc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:28:13 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:48408
I20260808 02:28:13.532366 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 4259
I20260808 02:28:13.544078 4743 tablet_replica.cc:330] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: stopping tablet replica
I20260808 02:28:13.544621 4743 raft_consensus.cc:2229] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:28:13.545288 4745 tablet_replica.cc:330] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: stopping tablet replica
I20260808 02:28:13.545727 4745 raft_consensus.cc:2229] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:28:13.546403 4746 tablet_replica.cc:330] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: stopping tablet replica
I20260808 02:28:13.546902 4746 raft_consensus.cc:2229] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:28:13.547672 4746 raft_consensus.cc:2258] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:28:13.548527 4745 raft_consensus.cc:2258] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:28:13.549060 4743 raft_consensus.cc:2258] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:28:13.552063 4745 ts_tablet_manager.cc:1905] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:28:13.555794 4743 ts_tablet_manager.cc:1905] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:28:13.556808 4746 ts_tablet_manager.cc:1905] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:28:13.564576 4746 ts_tablet_manager.cc:1918] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:28:13.564738 4746 log.cc:1196] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/7e5052d0319c4827a5d2d27e4be5dc34
I20260808 02:28:13.565280 4746 ts_tablet_manager.cc:1939] T 7e5052d0319c4827a5d2d27e4be5dc34 P b21c8493940d43baa9c6ff45b67b1556: Deleting consensus metadata
I20260808 02:28:13.566213 4743 ts_tablet_manager.cc:1918] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:28:13.566401 4743 log.cc:1196] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/7e5052d0319c4827a5d2d27e4be5dc34
I20260808 02:28:13.567777 4178 catalog_manager.cc:4841] TS b21c8493940d43baa9c6ff45b67b1556 (127.19.231.1:43323): tablet 7e5052d0319c4827a5d2d27e4be5dc34 (table test-table [id=755507a3df7b4abea1ec4449346b8627]) successfully deleted
I20260808 02:28:13.568810 4743 ts_tablet_manager.cc:1939] T 7e5052d0319c4827a5d2d27e4be5dc34 P 5e36bc5416f04b90b10c1bf4d0da03aa: Deleting consensus metadata
Aug 08 02:28:13 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:28:13.571170 4745 ts_tablet_manager.cc:1918] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:28:13.571338 4745 log.cc:1196] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/7e5052d0319c4827a5d2d27e4be5dc34
I20260808 02:28:13.571859 4745 ts_tablet_manager.cc:1939] T 7e5052d0319c4827a5d2d27e4be5dc34 P c39eebd671cc45559fa6705a71c27222: Deleting consensus metadata
I20260808 02:28:13.571740 4178 catalog_manager.cc:4841] TS 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445): tablet 7e5052d0319c4827a5d2d27e4be5dc34 (table test-table [id=755507a3df7b4abea1ec4449346b8627]) successfully deleted
I20260808 02:28:13.573596 4178 catalog_manager.cc:4841] TS c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377): tablet 7e5052d0319c4827a5d2d27e4be5dc34 (table test-table [id=755507a3df7b4abea1ec4449346b8627]) successfully deleted
I20260808 02:28:13.582131 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.533880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.19.231.1:43323)
0808 02:28:13.534112 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.534121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.553570 (+ 19449us) server_negotiation.cc:407] Connection header received
0808 02:28:13.553819 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.553826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.553980 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.554217 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:13.559891 (+ 5674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.561384 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.563843 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.564601 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.564707 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.573428 (+ 8721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:13.573446 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:13.573463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:13.573527 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:13.578366 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.579348 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.579357 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.579362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.579523 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.580027 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.580032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.580036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.580477 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:13.580803 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.581488 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.581742 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:28:13.584190 4393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:13.697597 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 4259
I20260808 02:28:13.722159 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 4405
Aug 08 02:28:13 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:28:13.738404 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.722977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.19.231.2:35445)
0808 02:28:13.723202 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.723208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.723328 (+ 120us) server_negotiation.cc:407] Connection header received
0808 02:28:13.723558 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.723563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.723641 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.723752 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:13.725210 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.725962 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.727458 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.727856 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.727995 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.732388 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:13.732404 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:13.732420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:13.732461 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:13.735432 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.736171 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.736178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.736181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.736304 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.736694 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.736698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.736703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.737120 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:13.737479 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.737780 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.738012 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260808 02:28:13.739109 4542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:13.830339 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 4405
I20260808 02:28:13.853858 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 4554
Aug 08 02:28:13 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:28:13.869344 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:13.854716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.19.231.3:42377)
0808 02:28:13.854949 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:13.854954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:13.854977 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:28:13.855188 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:13.855193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:13.855298 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:13.855395 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:13.856794 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.857513 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.858945 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:13.859329 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:13.859426 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:13.863949 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:13.863967 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:13.863984 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:13.864020 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:13.866583 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.867442 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.867448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.867451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.867594 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:13.867923 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:13.867926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:13.867928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:13.868268 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:13.868484 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:13.868769 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 02:28:13.868933 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:28:13.870030 4688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:13.960006 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 4554
I20260808 02:28:13.982251 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 4159
I20260808 02:28:13.983541 4225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:14.099357 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 4159
I20260808 02:28:34.127929 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43731
--webserver_interface=127.19.231.62
--webserver_port=40031
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:28:34.482275 4753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:34.482579 4753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:34.482793 4753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:34.490137 4753 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:28:34.490233 4753 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:28:34.490314 4753 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:28:34.490394 4753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:28:34.490489 4753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:34.490643 4753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:34.490720 4753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:28:34.490803 4753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:28:34.500998 4753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=40031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:34.502388 4753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:34.503836 4753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:34.532061 4760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:34.532025 4763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:34.532100 4759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:34.533102 4753 server_base.cc:1034] running on GCE node
I20260808 02:28:34.534004 4753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:34.535753 4753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:34.537026 4753 hybrid_clock.cc:648] HybridClock initialized: now 1786156114536911 us; error 65 us; skew 500 ppm
I20260808 02:28:34.537461 4753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:34 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156114, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:34.541914 4753 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:28:34.544114 4753 webserver.cc:460] Webserver started at http://127.19.231.62:40031/ using document root <none> and password file <none>
I20260808 02:28:34.544981 4753 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:34.545138 4753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:34.551497 4753 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 02:28:34.554483 4770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:34.555930 4753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:28:34.556124 4753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "be34fb3fd0ec42bcaf21c5cceac254e0"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "2b117df7d8144eb75f85ed4ec440dbf5"
server_key_iv: "db4214085df149ec067340e4d7e3e3e7"
server_key_version: "encryptionkey@0"
I20260808 02:28:34.557225 4753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:34.584059 4753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:34.588423 4753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:34.588745 4753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:34.608238 4753 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43731
I20260808 02:28:34.608273 4825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43731 every 8 connection(s)
I20260808 02:28:34.609800 4753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:28:34.611318 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 4753
I20260808 02:28:34.612430 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:43323
--local_ip_for_outbound_sockets=127.19.231.1
--tserver_master_addrs=127.19.231.62:43731
--webserver_port=46635
--webserver_interface=127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:28:34.620432 4826 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:28:34.625792 4826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Bootstrap starting.
I20260808 02:28:34.650595 4826 log.cc:824] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:34.663901 4826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:28:34.664516 4826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Bootstrap complete.
I20260808 02:28:34.674530 4826 raft_consensus.cc:348] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:34.675448 4826 raft_consensus.cc:729] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be34fb3fd0ec42bcaf21c5cceac254e0, State: Initialized, Role: FOLLOWER
I20260808 02:28:34.676205 4826 consensus_queue.cc:260] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:34.676445 4826 raft_consensus.cc:388] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:28:34.676569 4826 raft_consensus.cc:482] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:28:34.676764 4826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 02:28:34.680735 4826 raft_consensus.cc:504] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:34.681300 4826 leader_election.cc:302] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be34fb3fd0ec42bcaf21c5cceac254e0; no voters:
I20260808 02:28:34.681990 4826 leader_election.cc:288] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 02:28:34.682140 4831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 02:28:34.683507 4831 raft_consensus.cc:686] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 LEADER]: Becoming Leader. State: Replica: be34fb3fd0ec42bcaf21c5cceac254e0, State: Running, Role: LEADER
I20260808 02:28:34.684237 4831 consensus_queue.cc:237] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:34.684571 4826 sys_catalog.cc:564] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:28:34.686866 4833 sys_catalog.cc:455] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be34fb3fd0ec42bcaf21c5cceac254e0. Latest consensus state: current_term: 2 leader_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } } }
I20260808 02:28:34.687317 4833 sys_catalog.cc:458] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:28:34.687288 4832 sys_catalog.cc:455] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } } }
I20260808 02:28:34.688103 4832 sys_catalog.cc:458] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:28:34.697063 4840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:28:34.711637 4840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:28:34.714319 4840 catalog_manager.cc:1259] Loaded cluster ID: 0f3947d710bb454bb5b8f2c3a1ae576c
I20260808 02:28:34.714457 4840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:28:34.719554 4840 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:28:34.721251 4840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Loaded TSK: 0
I20260808 02:28:34.768523 4840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Generated new TSK 1
I20260808 02:28:34.769157 4840 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 02:28:35.008613 4828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:35.008978 4828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:35.009161 4828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:35.016866 4828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:35.017191 4828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:35.017405 4828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:28:35.028239 4828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:43323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=46635
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:35.029675 4828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:35.031195 4828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:35.045356 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:35.045709 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:35.046264 4859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:35.046581 4828 server_base.cc:1034] running on GCE node
I20260808 02:28:35.047292 4828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:35.048359 4828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:35.049638 4828 hybrid_clock.cc:648] HybridClock initialized: now 1786156115049557 us; error 70 us; skew 500 ppm
I20260808 02:28:35.049998 4828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:35 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:35.053418 4828 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:28:35.055420 4828 webserver.cc:460] Webserver started at http://127.19.231.1:46635/ using document root <none> and password file <none>
I20260808 02:28:35.056229 4828 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:35.056383 4828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:35.062398 4828 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:28:35.065411 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:35.066974 4828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:28:35.067164 4828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "b21c8493940d43baa9c6ff45b67b1556"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "4f976887f0398064fc8ea86a3e43a693"
server_key_iv: "eacf72fea3ff6f144a9fa907ed026941"
server_key_version: "encryptionkey@0"
I20260808 02:28:35.068199 4828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:35.091499 4828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:35.096141 4828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:35.096519 4828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:35.097689 4828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:35.099218 4828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:35.099325 4828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:35.099483 4828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:35.099593 4828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:35.135597 4828 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:43323
I20260808 02:28:35.135718 4985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:43323 every 8 connection(s)
I20260808 02:28:35.137144 4828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:28:35.142242 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 4828
I20260808 02:28:35.142951 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:35445
--local_ip_for_outbound_sockets=127.19.231.2
--tserver_master_addrs=127.19.231.62:43731
--webserver_port=33003
--webserver_interface=127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:28:35 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:28:35.163285 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:35.140143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:36743 (local address 127.19.231.62:43731)
0808 02:28:35.140871 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:35.140889 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:35.141448 (+ 559us) server_negotiation.cc:407] Connection header received
0808 02:28:35.144002 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:35.144023 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:35.144523 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:35.144923 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:35.146566 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.148589 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.150185 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.151384 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.151586 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:35.155269 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:35.155282 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:35.155304 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:35.155360 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:35.158496 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:35.159490 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:35.159496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:35.159502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:35.159674 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:35.160190 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:35.160193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:35.160195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:35.160640 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:35.160830 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:35.161794 (+ 964us) server_negotiation.cc:299] Negotiation successful
0808 02:28:35.162258 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":324,"threads_started":1}
I20260808 02:28:35.163717 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:35.140379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:36743)
0808 02:28:35.141264 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:35.141316 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:35.143384 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:35.145103 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:35.145111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:35.145702 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:28:35.146256 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:35.146275 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.148771 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.148775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:35.150009 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:35.150022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.151644 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.151649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:35.151837 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:35.152644 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:28:35.152675 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:28:35.154980 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:28:35.158674 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:35.158685 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:35.158702 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:35.159347 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:35.159801 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:35.159809 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:35.159816 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:35.160068 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:35.160800 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:28:35.160811 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:28:35.161274 (+ 463us) client_negotiation.cc:769] Sending connection context
0808 02:28:35.161597 (+ 323us) client_negotiation.cc:240] Negotiation successful
0808 02:28:35.162190 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":400,"threads_started":1}
I20260808 02:28:35.166539 4986 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43731
I20260808 02:28:35.166883 4986 heartbeater.cc:464] Registering TS with master...
I20260808 02:28:35.167728 4986 heartbeater.cc:511] Master 127.19.231.62:43731 requested a full tablet report, sending...
I20260808 02:28:35.170070 4787 ts_manager.cc:194] Registered new tserver with Master: b21c8493940d43baa9c6ff45b67b1556 (127.19.231.1:43323)
I20260808 02:28:35.171631 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:36743
W20260808 02:28:35.506182 4991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:35.506525 4991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:35.506724 4991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:35.513939 4991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:35.514097 4991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:35.514286 4991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:28:35.524224 4991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:35445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=33003
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:35.525624 4991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:35.527072 4991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:35.540427 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:35.540504 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:35.540470 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:35.541635 4991 server_base.cc:1034] running on GCE node
I20260808 02:28:35.542193 4991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:35.543140 4991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:35.544381 4991 hybrid_clock.cc:648] HybridClock initialized: now 1786156115544333 us; error 55 us; skew 500 ppm
I20260808 02:28:35.544713 4991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:35 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:35.547950 4991 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:28:35.549460 4991 webserver.cc:460] Webserver started at http://127.19.231.2:33003/ using document root <none> and password file <none>
I20260808 02:28:35.550208 4991 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:35.550380 4991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:35.556102 4991 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 02:28:35.559152 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:35.560591 4991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:28:35.560770 4991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "5e36bc5416f04b90b10c1bf4d0da03aa"
format_stamp: "Formatted at 2026-08-08 02:28:12 on dist-test-slave-pgkr"
server_key: "db47ceb460225e22bf5539d1fd68643c"
server_key_iv: "5dc6f966f94dea88977a75eaf30c3d13"
server_key_version: "encryptionkey@0"
I20260808 02:28:35.561870 4991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:35.580547 4991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:35.584693 4991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:35.585006 4991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:35.586127 4991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:35.587641 4991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:35.587770 4991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:35.587909 4991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:35.587999 4991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:35.625193 4991 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:35445
I20260808 02:28:35.625244 5134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:35445 every 8 connection(s)
I20260808 02:28:35.626744 4991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:28:35.628575 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 4991
I20260808 02:28:35.629323 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:42377
--local_ip_for_outbound_sockets=127.19.231.3
--tserver_master_addrs=127.19.231.62:43731
--webserver_port=38611
--webserver_interface=127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 02:28:35 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156115, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:28:35.651820 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:35.631812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:54127 (local address 127.19.231.62:43731)
0808 02:28:35.632078 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:35.632088 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:35.632937 (+ 849us) server_negotiation.cc:407] Connection header received
0808 02:28:35.634937 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:35.634946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:35.635088 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:35.635214 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:35.636925 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.638054 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.639983 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.640716 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.640854 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:35.645492 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:35.645497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:35.645500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:35.645534 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:35.648386 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:35.649385 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:35.649389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:35.649391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:35.649519 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:35.650047 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:35.650051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:35.650053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:35.650337 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:35.650449 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:35.651385 (+ 936us) server_negotiation.cc:299] Negotiation successful
0808 02:28:35.651568 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 02:28:35.653566 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:35.630503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:54127)
0808 02:28:35.632742 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:35.632790 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:35.634584 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:35.635435 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:35.635442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:35.636100 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:28:35.636742 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:35.636761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.638268 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.638272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:35.639747 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:35.639760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:35.640975 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:35.640984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:35.641311 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:35.642093 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:28:35.642124 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:28:35.645246 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:28:35.648561 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:35.648574 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:35.648594 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:35.649260 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:35.649633 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:35.649640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:35.649647 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:35.649924 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:35.650473 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:28:35.650484 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:28:35.650983 (+ 499us) client_negotiation.cc:769] Sending connection context
0808 02:28:35.651310 (+ 327us) client_negotiation.cc:240] Negotiation successful
0808 02:28:35.651910 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1906,"thread_start_us":1650,"threads_started":1}
I20260808 02:28:35.655202 5135 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43731
I20260808 02:28:35.655534 5135 heartbeater.cc:464] Registering TS with master...
I20260808 02:28:35.656286 5135 heartbeater.cc:511] Master 127.19.231.62:43731 requested a full tablet report, sending...
I20260808 02:28:35.657784 4787 ts_manager.cc:194] Registered new tserver with Master: 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445)
I20260808 02:28:35.658748 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:54127
W20260808 02:28:35.992228 5138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:35.992543 5138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:35.992733 5138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:36.000890 5138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:36.001027 5138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:36.001223 5138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:28:36.011260 5138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:42377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=38611
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:36.012650 5138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:36.013999 5138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:36.027318 5144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:36.027388 5145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:36.027407 5147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:36.028381 5138 server_base.cc:1034] running on GCE node
I20260808 02:28:36.029019 5138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:36.029949 5138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:36.031203 5138 hybrid_clock.cc:648] HybridClock initialized: now 1786156116031132 us; error 71 us; skew 500 ppm
I20260808 02:28:36.031526 5138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:36 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156116, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:36.034931 5138 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:28:36.036507 5138 webserver.cc:460] Webserver started at http://127.19.231.3:38611/ using document root <none> and password file <none>
I20260808 02:28:36.037240 5138 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:36.037415 5138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:36.043249 5138 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:28:36.046141 5154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:36.047590 5138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:28:36.047802 5138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "c39eebd671cc45559fa6705a71c27222"
format_stamp: "Formatted at 2026-08-08 02:28:13 on dist-test-slave-pgkr"
server_key: "4452f183100bbffd3528d8dfc218066c"
server_key_iv: "6cea29e13644d8f9aa47634e3fb8b7df"
server_key_version: "encryptionkey@0"
I20260808 02:28:36.048784 5138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:36.073850 5138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:36.078159 5138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:36.078450 5138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:36.079588 5138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:36.080986 5138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:36.081141 5138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:36.081271 5138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:36.081362 5138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:36.117574 5138 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:42377
I20260808 02:28:36.117673 5277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:42377 every 8 connection(s)
I20260808 02:28:36.119124 5138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:28:36.127090 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 5138
Aug 08 02:28:36 dist-test-slave-pgkr krb5kdc[4146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156116, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:28:36.142191 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.121934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:43665 (local address 127.19.231.62:43731)
0808 02:28:36.122177 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.122183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.123473 (+ 1290us) server_negotiation.cc:407] Connection header received
0808 02:28:36.125304 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.125309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.125377 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.125470 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:36.127016 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.128104 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.130110 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.130514 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.130669 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.134679 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:36.134686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:36.134692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:36.134736 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:36.138695 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.139630 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.139635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.139639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.139859 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.140406 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.140412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.140415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.140711 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:36.140881 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.141604 (+ 723us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.141821 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:28:36.143332 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.122450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:43665)
0808 02:28:36.123311 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:36.123355 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:36.125021 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:36.125681 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:36.125689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:36.126213 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:28:36.126839 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.126859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.128279 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.128286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.129962 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.129975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.130784 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.130792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.131066 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.131840 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:28:36.131874 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:28:36.134408 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:28:36.138830 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:36.138839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:36.138853 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:36.139490 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:36.139997 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:28:36.140002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:28:36.140005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:28:36.140291 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:28:36.140847 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:28:36.140855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:28:36.141246 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 02:28:36.141502 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 02:28:36.142016 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":368,"threads_started":1}
I20260808 02:28:36.144627 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.128132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.19.231.62:43731)
0808 02:28:36.128647 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.128653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.129074 (+ 421us) server_negotiation.cc:407] Connection header received
0808 02:28:36.129215 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.129221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.129290 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.129440 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:36.131115 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.132146 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.133709 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.134366 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.134508 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.138204 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:36.138211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:36.138215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:36.138264 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:36.141715 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.142455 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.142459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.142462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.142601 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.143082 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.143085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.143087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.143345 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:36.143638 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.143997 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.144220 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
I20260808 02:28:36.144915 5278 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43731
I20260808 02:28:36.145248 5278 heartbeater.cc:464] Registering TS with master...
I20260808 02:28:36.146090 5278 heartbeater.cc:511] Master 127.19.231.62:43731 requested a full tablet report, sending...
I20260808 02:28:36.147591 4787 ts_manager.cc:194] Registered new tserver with Master: c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377)
I20260808 02:28:36.148342 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:43665
I20260808 02:28:36.159821 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:28:36.166978 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.160807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.19.231.62:43731)
0808 02:28:36.160986 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.160990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.161499 (+ 509us) server_negotiation.cc:407] Connection header received
0808 02:28:36.161771 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.161794 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.161964 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.162140 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:28:36.162580 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.163327 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.164813 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.165258 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.165390 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.165983 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:28:36.166267 (+ 284us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 02:28:36.166698 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43150: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 02:28:36.167130 5281 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43150: token expired
W20260808 02:28:36.167680 4700 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.231.62:43731)
I20260808 02:28:36.175289 4986 heartbeater.cc:503] Master 127.19.231.62:43731 was elected leader, sending a full tablet report...
I20260808 02:28:36.181151 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.168324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.19.231.62:43731)
0808 02:28:36.168510 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.168517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.168541 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:28:36.168590 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.168594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.168652 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.168748 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:36.170141 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.170970 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.172411 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.172868 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.172997 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.175873 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:36.175882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:36.175891 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:36.175924 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:36.178813 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.179482 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.179486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.179488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.179608 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.179931 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.179935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.179936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.180236 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:36.180412 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.180672 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.180837 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:28:36.184501 4787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:28:36.191865 4787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:28:36.215299 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.205532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:43323 (local address 127.0.0.1:48352)
0808 02:28:36.206147 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:36.206165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:36.206277 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:36.207567 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:36.207577 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:36.207601 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:36.208028 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.208045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.209918 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.209926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.212087 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.212098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.214032 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.214039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.214270 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.214337 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 02:28:36.214530 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 02:28:36.214754 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":129,"thread_start_us":346,"threads_started":1}
I20260808 02:28:36.216004 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.204593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.19.231.3:42377)
0808 02:28:36.205404 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.205414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.205450 (+ 36us) server_negotiation.cc:407] Connection header received
0808 02:28:36.205559 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.205564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.205877 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.206071 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:36.207222 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.208923 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.212325 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.213785 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.213923 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.214177 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.215459 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.215641 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":427,"threads_started":1}
I20260808 02:28:36.216261 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.204189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:42377 (local address 127.0.0.1:36696)
0808 02:28:36.205141 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:36.205189 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:36.205367 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:36.206495 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:36.206506 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:36.206545 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:36.206925 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.206935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.209065 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.209069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.212135 (+ 3066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.212155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.214339 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.214343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.214930 (+ 587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.214968 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:28:36.215059 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:28:36.215206 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":4,"thread_start_us":510,"threads_started":1}
I20260808 02:28:36.216302 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.205917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.19.231.1:43323)
0808 02:28:36.206888 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.206896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.206928 (+ 32us) server_negotiation.cc:407] Connection header received
0808 02:28:36.207046 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.207057 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.207325 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.207573 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:36.208247 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.209780 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.212250 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.213785 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.213923 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.214165 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.214578 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.214786 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":584,"threads_started":1}
I20260808 02:28:36.216842 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.206218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:35445 (local address 127.0.0.1:38386)
0808 02:28:36.208323 (+ 2105us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:36.208340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:36.208451 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:36.209119 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:36.209123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:36.209165 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:36.209598 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.209610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.211749 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.211755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.213410 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.213424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.215822 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.215830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.216031 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.216066 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:28:36.216183 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 02:28:36.216352 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"mutex_wait_us":1,"thread_start_us":1829,"threads_started":1}
I20260808 02:28:36.217012 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.206761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.19.231.2:35445)
0808 02:28:36.207391 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.207401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.208582 (+ 1181us) server_negotiation.cc:407] Connection header received
0808 02:28:36.208697 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.208708 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.208922 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.209132 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:36.209771 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.211539 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.213597 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.215121 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.215287 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.215841 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.216554 (+ 713us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.216715 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":318,"threads_started":1}
I20260808 02:28:36.220472 5205 tablet_service.cc:1467] Processing CreateTablet for tablet 7fca971af4f84416ad6dab97ee2ba0a6 (DEFAULT_TABLE table=test-table [id=b5f1bc0d675e467ea9569b2bb3c699fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:28:36.220719 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 7fca971af4f84416ad6dab97ee2ba0a6 (DEFAULT_TABLE table=test-table [id=b5f1bc0d675e467ea9569b2bb3c699fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:28:36.221856 5205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fca971af4f84416ad6dab97ee2ba0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:28:36.220794 5063 tablet_service.cc:1467] Processing CreateTablet for tablet 7fca971af4f84416ad6dab97ee2ba0a6 (DEFAULT_TABLE table=test-table [id=b5f1bc0d675e467ea9569b2bb3c699fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:28:36.222235 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fca971af4f84416ad6dab97ee2ba0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:28:36.223006 5063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fca971af4f84416ad6dab97ee2ba0a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:28:36.234105 5295 tablet_bootstrap.cc:492] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Bootstrap starting.
I20260808 02:28:36.237514 5295 tablet_bootstrap.cc:654] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Neither blocks nor log segments found. Creating new log.
I20260808 02:28:36.238837 5295 log.cc:824] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:36.239420 5296 tablet_bootstrap.cc:492] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Bootstrap starting.
I20260808 02:28:36.241753 5297 tablet_bootstrap.cc:492] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Bootstrap starting.
I20260808 02:28:36.243060 5295 tablet_bootstrap.cc:492] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: No bootstrap required, opened a new log
I20260808 02:28:36.243343 5295 ts_tablet_manager.cc:1397] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 02:28:36.243546 5296 tablet_bootstrap.cc:654] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Neither blocks nor log segments found. Creating new log.
I20260808 02:28:36.244894 5296 log.cc:824] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:36.245491 5297 tablet_bootstrap.cc:654] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Neither blocks nor log segments found. Creating new log.
I20260808 02:28:36.246886 5297 log.cc:824] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:36.248428 5296 tablet_bootstrap.cc:492] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: No bootstrap required, opened a new log
I20260808 02:28:36.248718 5296 ts_tablet_manager.cc:1397] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 02:28:36.250061 5297 tablet_bootstrap.cc:492] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: No bootstrap required, opened a new log
I20260808 02:28:36.250351 5297 ts_tablet_manager.cc:1397] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260808 02:28:36.252162 5295 raft_consensus.cc:348] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.252715 5295 raft_consensus.cc:374] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:28:36.252842 5295 raft_consensus.cc:729] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39eebd671cc45559fa6705a71c27222, State: Initialized, Role: FOLLOWER
I20260808 02:28:36.253511 5295 consensus_queue.cc:260] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.254796 5278 heartbeater.cc:503] Master 127.19.231.62:43731 was elected leader, sending a full tablet report...
I20260808 02:28:36.255300 5295 ts_tablet_manager.cc:1428] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260808 02:28:36.260874 5296 raft_consensus.cc:348] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.261042 5297 raft_consensus.cc:348] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.261363 5296 raft_consensus.cc:374] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:28:36.261451 5297 raft_consensus.cc:374] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:28:36.261478 5296 raft_consensus.cc:729] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36bc5416f04b90b10c1bf4d0da03aa, State: Initialized, Role: FOLLOWER
I20260808 02:28:36.261544 5297 raft_consensus.cc:729] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21c8493940d43baa9c6ff45b67b1556, State: Initialized, Role: FOLLOWER
I20260808 02:28:36.262214 5297 consensus_queue.cc:260] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.262213 5296 consensus_queue.cc:260] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.263892 5135 heartbeater.cc:503] Master 127.19.231.62:43731 was elected leader, sending a full tablet report...
I20260808 02:28:36.264230 5297 ts_tablet_manager.cc:1428] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260808 02:28:36.264472 5296 ts_tablet_manager.cc:1428] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260808 02:28:36.370906 5279 tablet.cc:2369] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:28:36.379354 5136 tablet.cc:2369] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:28:36.389724 4987 tablet.cc:2369] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:28:36.609155 5302 raft_consensus.cc:482] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:28:36.609514 5302 raft_consensus.cc:504] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.611447 5302 leader_election.cc:288] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377), 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445)
I20260808 02:28:36.619885 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.611904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:44441 (local address 127.19.231.3:42377)
0808 02:28:36.612094 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.612099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.612772 (+ 673us) server_negotiation.cc:407] Connection header received
0808 02:28:36.613099 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.613105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.613189 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.613261 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:36.613946 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.615035 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.618000 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.618954 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.619155 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.619230 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.619500 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.619628 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:28:36.619940 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.611930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:42377 (local address 127.19.231.1:44441)
0808 02:28:36.612626 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:36.612643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:36.612913 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:36.613475 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:36.613478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:36.613500 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:36.613801 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.613808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.615218 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.615228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.617819 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.617836 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.619172 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.619178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.619335 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.619358 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 02:28:36.619462 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:28:36.619599 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":368,"threads_started":1}
I20260808 02:28:36.621019 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.613092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:59447 (local address 127.19.231.2:35445)
0808 02:28:36.613303 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.613310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.614503 (+ 1193us) server_negotiation.cc:407] Connection header received
0808 02:28:36.614805 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.614811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.614899 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.615015 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:28:36.615763 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.616464 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.618280 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.619671 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.619835 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.619911 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.620620 (+ 709us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.620765 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:28:36.621099 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.612769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:35445 (local address 127.19.231.1:59447)
0808 02:28:36.614384 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:36.614404 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:28:36.614583 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:28:36.615044 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:28:36.615048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:28:36.615083 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:28:36.615487 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.615496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.616599 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.616606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.618096 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:28:36.618125 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.620249 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.620256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:28:36.620442 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.620478 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:28:36.620573 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 02:28:36.620709 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"mutex_wait_us":6,"thread_start_us":1296,"threads_started":1}
I20260808 02:28:36.621168 5227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fca971af4f84416ad6dab97ee2ba0a6" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39eebd671cc45559fa6705a71c27222" is_pre_election: true
I20260808 02:28:36.621721 5227 raft_consensus.cc:2454] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 0.
I20260808 02:28:36.622252 5084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fca971af4f84416ad6dab97ee2ba0a6" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" is_pre_election: true
I20260808 02:28:36.622464 4868 leader_election.cc:302] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b21c8493940d43baa9c6ff45b67b1556, c39eebd671cc45559fa6705a71c27222; no voters:
I20260808 02:28:36.622884 5302 raft_consensus.cc:2781] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:28:36.622824 5084 raft_consensus.cc:2454] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 0.
I20260808 02:28:36.623004 5302 raft_consensus.cc:482] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:28:36.623070 5302 raft_consensus.cc:3037] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:28:36.626230 5302 raft_consensus.cc:504] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.627081 5302 leader_election.cc:288] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 election: Requested vote from peers c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377), 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445)
I20260808 02:28:36.627573 5227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fca971af4f84416ad6dab97ee2ba0a6" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39eebd671cc45559fa6705a71c27222"
I20260808 02:28:36.627771 5227 raft_consensus.cc:3037] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:28:36.627908 5084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fca971af4f84416ad6dab97ee2ba0a6" candidate_uuid: "b21c8493940d43baa9c6ff45b67b1556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa"
I20260808 02:28:36.628111 5084 raft_consensus.cc:3037] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:28:36.630895 5084 raft_consensus.cc:2454] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 1.
I20260808 02:28:36.631012 5227 raft_consensus.cc:2454] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b21c8493940d43baa9c6ff45b67b1556 in term 1.
I20260808 02:28:36.631459 4868 leader_election.cc:302] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e36bc5416f04b90b10c1bf4d0da03aa, b21c8493940d43baa9c6ff45b67b1556; no voters:
I20260808 02:28:36.631788 5302 raft_consensus.cc:2781] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:28:36.632273 5302 raft_consensus.cc:686] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 1 LEADER]: Becoming Leader. State: Replica: b21c8493940d43baa9c6ff45b67b1556, State: Running, Role: LEADER
I20260808 02:28:36.632797 5302 consensus_queue.cc:237] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } }
I20260808 02:28:36.639115 4787 catalog_manager.cc:5495] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 reported cstate change: term changed from 0 to 1, leader changed from <none> to b21c8493940d43baa9c6ff45b67b1556 (127.19.231.1). New cstate: current_term: 1 leader_uuid: "b21c8493940d43baa9c6ff45b67b1556" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b21c8493940d43baa9c6ff45b67b1556" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 43323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 } health_report { overall_health: UNKNOWN } } }
I20260808 02:28:36.673431 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.667696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.19.231.1:43323)
0808 02:28:36.667934 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.667941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.667962 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:28:36.668211 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.668215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.668296 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.668451 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:28:36.668848 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.669629 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.671144 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.671547 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.671694 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.672292 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:28:36.672808 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:28:36.672972 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.673106 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.673175 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:28:36.681912 5227 raft_consensus.cc:1261] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Refusing update from remote peer b21c8493940d43baa9c6ff45b67b1556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:28:36.681907 5084 raft_consensus.cc:1261] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Refusing update from remote peer b21c8493940d43baa9c6ff45b67b1556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:28:36.682991 5307 consensus_queue.cc:1035] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e36bc5416f04b90b10c1bf4d0da03aa" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:28:36.683391 5302 consensus_queue.cc:1035] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39eebd671cc45559fa6705a71c27222" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:28:36.693723 5311 mvcc.cc:205] Tried to move back new op lower bound from 7316095453918314496 to 7316095453732487168. Current Snapshot: MvccSnapshot[applied={T|T < 7316095453918314496}]
I20260808 02:28:36.695720 5312 mvcc.cc:205] Tried to move back new op lower bound from 7316095453918314496 to 7316095453732487168. Current Snapshot: MvccSnapshot[applied={T|T < 7316095453918314496}]
I20260808 02:28:36.714347 4787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:28:36.714689 4787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:28:36.718015 4787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Sending DeleteTablet for 3 replicas of tablet 7fca971af4f84416ad6dab97ee2ba0a6
I20260808 02:28:36.719383 5205 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fca971af4f84416ad6dab97ee2ba0a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:28:36 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:36696
I20260808 02:28:36.719622 4916 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fca971af4f84416ad6dab97ee2ba0a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:28:36 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:48352
I20260808 02:28:36.719714 5063 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fca971af4f84416ad6dab97ee2ba0a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:28:36 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:38386
I20260808 02:28:36.721009 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 4828
I20260808 02:28:36.729617 5322 tablet_replica.cc:330] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: stopping tablet replica
I20260808 02:28:36.730262 5322 raft_consensus.cc:2229] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:28:36.731113 5322 raft_consensus.cc:2258] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:28:36.733572 5322 ts_tablet_manager.cc:1905] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:28:36.734241 5321 tablet_replica.cc:330] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: stopping tablet replica
I20260808 02:28:36.734763 5321 raft_consensus.cc:2229] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:28:36.735313 5323 tablet_replica.cc:330] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: stopping tablet replica
I20260808 02:28:36.735803 5323 raft_consensus.cc:2229] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:28:36.735352 5321 raft_consensus.cc:2258] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:28:36.736388 5323 raft_consensus.cc:2258] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:28:36.738912 5323 ts_tablet_manager.cc:1905] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:28:36.741550 5321 ts_tablet_manager.cc:1905] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:28:36.746117 5323 ts_tablet_manager.cc:1918] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:28:36.746325 5323 log.cc:1196] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/7fca971af4f84416ad6dab97ee2ba0a6
I20260808 02:28:36.746973 5323 ts_tablet_manager.cc:1939] T 7fca971af4f84416ad6dab97ee2ba0a6 P 5e36bc5416f04b90b10c1bf4d0da03aa: Deleting consensus metadata
I20260808 02:28:36.746945 5322 ts_tablet_manager.cc:1918] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:28:36.747125 5322 log.cc:1196] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/7fca971af4f84416ad6dab97ee2ba0a6
I20260808 02:28:36.747781 5322 ts_tablet_manager.cc:1939] T 7fca971af4f84416ad6dab97ee2ba0a6 P b21c8493940d43baa9c6ff45b67b1556: Deleting consensus metadata
I20260808 02:28:36.749367 4772 catalog_manager.cc:4841] TS 5e36bc5416f04b90b10c1bf4d0da03aa (127.19.231.2:35445): tablet 7fca971af4f84416ad6dab97ee2ba0a6 (table test-table [id=b5f1bc0d675e467ea9569b2bb3c699fd]) successfully deleted
I20260808 02:28:36.749768 4772 catalog_manager.cc:4841] TS b21c8493940d43baa9c6ff45b67b1556 (127.19.231.1:43323): tablet 7fca971af4f84416ad6dab97ee2ba0a6 (table test-table [id=b5f1bc0d675e467ea9569b2bb3c699fd]) successfully deleted
I20260808 02:28:36.753139 5321 ts_tablet_manager.cc:1918] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:28:36.753346 5321 log.cc:1196] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/7fca971af4f84416ad6dab97ee2ba0a6
I20260808 02:28:36.753898 5321 ts_tablet_manager.cc:1939] T 7fca971af4f84416ad6dab97ee2ba0a6 P c39eebd671cc45559fa6705a71c27222: Deleting consensus metadata
I20260808 02:28:36.755682 4772 catalog_manager.cc:4841] TS c39eebd671cc45559fa6705a71c27222 (127.19.231.3:42377): tablet 7fca971af4f84416ad6dab97ee2ba0a6 (table test-table [id=b5f1bc0d675e467ea9569b2bb3c699fd]) successfully deleted
I20260808 02:28:36.758088 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.722415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.19.231.1:43323)
0808 02:28:36.722663 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.722672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.734603 (+ 11931us) server_negotiation.cc:407] Connection header received
0808 02:28:36.734869 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.734876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.734989 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.735091 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:36.737451 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.738846 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.740967 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.741681 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.741782 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.748076 (+ 6294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:36.748094 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:36.748118 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:36.748175 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:36.753548 (+ 5373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.754587 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.754596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.754600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.754832 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.755330 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.755335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.755338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.755829 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:36.756131 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.757563 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.757770 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:28:36.758994 4981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:36.859867 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 4828
I20260808 02:28:36.885793 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 4991
I20260808 02:28:36.899606 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:36.886594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.19.231.2:35445)
0808 02:28:36.886793 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:36.886797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:36.886817 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:28:36.886980 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:36.886984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:36.887049 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:36.887141 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:36.888557 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.889315 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.890819 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:36.891211 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:36.891307 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:36.893908 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:36.893922 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:36.893937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:36.893976 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:36.896878 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.897586 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.897594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.897596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.897756 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:36.898123 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:36.898126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:36.898128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:36.898499 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:36.898800 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:36.899056 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 02:28:36.899225 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 02:28:36.900365 5130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:36.991250 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 4991
I20260808 02:28:37.014222 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 5138
I20260808 02:28:37.028607 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:37.015150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.19.231.3:42377)
0808 02:28:37.015343 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:28:37.015350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:28:37.015446 (+ 96us) server_negotiation.cc:407] Connection header received
0808 02:28:37.015727 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:28:37.015731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:28:37.015801 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:28:37.015916 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:28:37.017354 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:37.018075 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:37.019411 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:28:37.019813 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:28:37.019949 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:28:37.022603 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:28:37.022669 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:28:37.022691 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:28:37.022739 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:28:37.025550 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:37.026293 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:37.026306 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:37.026309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:37.026455 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:28:37.027008 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:28:37.027013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:28:37.027018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:28:37.027382 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:28:37.027703 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 02:28:37.028019 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 02:28:37.028244 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:28:37.029310 5271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:37.119915 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 5138
I20260808 02:28:37.138708 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 4753
I20260808 02:28:37.140064 4820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:28:37.261283 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 4753
I20260808 02:28:57.290171 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43731
--webserver_interface=127.19.231.62
--webserver_port=40031
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:28:57.657137 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:57.657423 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:57.657595 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:57.665025 5330 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 02:28:57.665125 5330 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 02:28:57.665165 5330 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 02:28:57.665213 5330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:28:57.665297 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:57.665432 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:57.665515 5330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:28:57.665642 5330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:28:57.675850 5330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=40031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:57.677248 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:57.678670 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:57.690856 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:57.690896 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:57.690958 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:57.692588 5330 server_base.cc:1034] running on GCE node
I20260808 02:28:57.693310 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:57.695030 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:57.696400 5330 hybrid_clock.cc:648] HybridClock initialized: now 1786156137696301 us; error 81 us; skew 500 ppm
I20260808 02:28:57.696854 5330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:57 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156137, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:57.700392 5330 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:28:57.702332 5330 webserver.cc:460] Webserver started at http://127.19.231.62:40031/ using document root <none> and password file <none>
I20260808 02:28:57.703135 5330 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:57.703255 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:57.708907 5330 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 02:28:57.711936 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:57.713361 5330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:28:57.713621 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "be34fb3fd0ec42bcaf21c5cceac254e0"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "2b117df7d8144eb75f85ed4ec440dbf5"
server_key_iv: "db4214085df149ec067340e4d7e3e3e7"
server_key_version: "encryptionkey@0"
I20260808 02:28:57.715011 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:57.742063 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:57.746591 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:57.746935 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:57.766518 5330 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43731
I20260808 02:28:57.766552 5405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43731 every 8 connection(s)
I20260808 02:28:57.768138 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:28:57.773350 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 5330
I20260808 02:28:57.774212 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:43323
--local_ip_for_outbound_sockets=127.19.231.1
--tserver_master_addrs=127.19.231.62:43731
--webserver_port=46635
--webserver_interface=127.19.231.1
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:28:57.775440 5406 sys_catalog.cc:263] Verifying existing consensus state
I20260808 02:28:57.778769 5406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Bootstrap starting.
I20260808 02:28:57.800216 5406 log.cc:824] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:28:57.816836 5406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 02:28:57.817414 5406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Bootstrap complete.
I20260808 02:28:57.827407 5406 raft_consensus.cc:348] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:57.828372 5406 raft_consensus.cc:729] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: be34fb3fd0ec42bcaf21c5cceac254e0, State: Initialized, Role: FOLLOWER
I20260808 02:28:57.829074 5406 consensus_queue.cc:260] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:57.829347 5406 raft_consensus.cc:388] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:28:57.829481 5406 raft_consensus.cc:482] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:28:57.829664 5406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 02:28:57.833675 5406 raft_consensus.cc:504] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:57.834192 5406 leader_election.cc:302] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be34fb3fd0ec42bcaf21c5cceac254e0; no voters:
I20260808 02:28:57.834934 5406 leader_election.cc:288] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 02:28:57.835078 5411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 02:28:57.836488 5411 raft_consensus.cc:686] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [term 3 LEADER]: Becoming Leader. State: Replica: be34fb3fd0ec42bcaf21c5cceac254e0, State: Running, Role: LEADER
I20260808 02:28:57.837073 5411 consensus_queue.cc:237] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } }
I20260808 02:28:57.837378 5406 sys_catalog.cc:564] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:28:57.839877 5413 sys_catalog.cc:455] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be34fb3fd0ec42bcaf21c5cceac254e0. Latest consensus state: current_term: 3 leader_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } } }
I20260808 02:28:57.840912 5413 sys_catalog.cc:458] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:28:57.840333 5412 sys_catalog.cc:455] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be34fb3fd0ec42bcaf21c5cceac254e0" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43731 } } }
I20260808 02:28:57.841445 5412 sys_catalog.cc:458] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0 [sys.catalog]: This master's current role is: LEADER
I20260808 02:28:57.843890 5417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:28:57.857646 5417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:28:57.859727 5417 catalog_manager.cc:1259] Loaded cluster ID: 0f3947d710bb454bb5b8f2c3a1ae576c
I20260808 02:28:57.859937 5417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:28:57.865406 5417 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:28:57.867673 5417 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Loaded TSK: 0
I20260808 02:28:57.867971 5417 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be34fb3fd0ec42bcaf21c5cceac254e0: Loaded TSK: 1
F20260808 02:28:57.869556 5417 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd68d2ff9ad google::LogMessage::Fail() at ??:0
@ 0x7fd68d30288a google::LogMessage::SendToLog() at ??:0
@ 0x7fd68d2ff4de google::LogMessage::Flush() at ??:0
@ 0x7fd68d300a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd6910637d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd69105273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd6a2d321c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd6a2d31783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd6a2dbd2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd6a2dbd0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd6a2e075e9 std::function<>::operator()() at ??:0
@ 0x7fd6a2d35cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd6a2d33c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd6a2dbb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd6a2dbb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd68ff4b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd68ff4e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd68ff4e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd68ff29eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd6932036db start_thread at ??:0
@ 0x7fd68c19961f clone at ??:0
W20260808 02:28:58.196408 5409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:58.196760 5409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:58.196942 5409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:58.204531 5409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:58.204696 5409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:58.204886 5409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:28:58.215397 5409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:43323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=46635
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:58.216799 5409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:58.218194 5409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:58.232332 5459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:58.232271 5461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:58.232337 5458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:58.233155 5409 server_base.cc:1034] running on GCE node
I20260808 02:28:58.233737 5409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:58.234730 5409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:58.235949 5409 hybrid_clock.cc:648] HybridClock initialized: now 1786156138235907 us; error 33 us; skew 500 ppm
I20260808 02:28:58.236312 5409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:58 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156138, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:58.239676 5409 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:28:58.241312 5409 webserver.cc:460] Webserver started at http://127.19.231.1:46635/ using document root <none> and password file <none>
I20260808 02:28:58.242051 5409 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:58.242188 5409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:58.247933 5409 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:28:58.250854 5468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:58.252390 5409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:28:58.252578 5409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "b21c8493940d43baa9c6ff45b67b1556"
format_stamp: "Formatted at 2026-08-08 02:28:11 on dist-test-slave-pgkr"
server_key: "4f976887f0398064fc8ea86a3e43a693"
server_key_iv: "eacf72fea3ff6f144a9fa907ed026941"
server_key_version: "encryptionkey@0"
I20260808 02:28:58.253656 5409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:58.277730 5409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:58.282286 5409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:58.282662 5409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:58.283766 5409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:58.285246 5409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:58.285386 5409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:58.285517 5409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:58.285622 5409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:58.324298 5409 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:43323
I20260808 02:28:58.324389 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:43323 every 8 connection(s)
I20260808 02:28:58.325944 5409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:28:58.327751 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 5409
I20260808 02:28:58.328491 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:35445
--local_ip_for_outbound_sockets=127.19.231.2
--tserver_master_addrs=127.19.231.62:43731
--webserver_port=33003
--webserver_interface=127.19.231.2
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:28:58.337007 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:58.329260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:32981)
0808 02:28:58.335381 (+ 6121us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:58.335725 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5738,"thread_start_us":5506,"threads_started":1}
W20260808 02:28:58.337816 5590 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:43731 (0 consecutive failures): Network error: Failed to ping master at 127.19.231.62:43731: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
I20260808 02:28:58.338881 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:58.338382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:51459)
0808 02:28:58.338588 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:58.338713 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:28:58.339902 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:58.339452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:46955)
0808 02:28:58.339618 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:58.339724 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:28:58.342051 5590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:28:58.694295 5594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:58.694715 5594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:58.694907 5594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:58.703267 5594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:58.703430 5594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:58.703676 5594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:28:58.713680 5594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:35445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=33003
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:58.715060 5594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:58.716400 5594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:58.731808 5600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:58.731786 5602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:58.732028 5599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:58.736275 5594 server_base.cc:1034] running on GCE node
I20260808 02:28:58.737006 5594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:58.738011 5594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:58.739225 5594 hybrid_clock.cc:648] HybridClock initialized: now 1786156138739167 us; error 37 us; skew 500 ppm
I20260808 02:28:58.739565 5594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:58 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156138, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:58.743124 5594 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:28:58.744681 5594 webserver.cc:460] Webserver started at http://127.19.231.2:33003/ using document root <none> and password file <none>
I20260808 02:28:58.745504 5594 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:58.745674 5594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:58.751857 5594 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 02:28:58.754977 5609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:58.756443 5594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:28:58.756628 5594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "5e36bc5416f04b90b10c1bf4d0da03aa"
format_stamp: "Formatted at 2026-08-08 02:28:12 on dist-test-slave-pgkr"
server_key: "db47ceb460225e22bf5539d1fd68643c"
server_key_iv: "5dc6f966f94dea88977a75eaf30c3d13"
server_key_version: "encryptionkey@0"
I20260808 02:28:58.757853 5594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:58.779054 5594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:58.783568 5594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:58.783857 5594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:58.784957 5594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:58.786384 5594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:58.786473 5594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:58.786644 5594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:58.786749 5594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 02:28:58.826428 5594 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:35445
I20260808 02:28:58.826500 5731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:35445 every 8 connection(s)
I20260808 02:28:58.828248 5594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:28:58.833804 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 5594
I20260808 02:28:58.834481 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:58.831871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:45427)
0808 02:28:58.832766 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:58.833052 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":379,"threads_started":1}
I20260808 02:28:58.834827 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:42377
--local_ip_for_outbound_sockets=127.19.231.3
--tserver_master_addrs=127.19.231.62:43731
--webserver_port=38611
--webserver_interface=127.19.231.3
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:28:58.835306 5732 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:43731 (0 consecutive failures): Network error: Failed to ping master at 127.19.231.62:43731: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
I20260808 02:28:58.836354 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:58.835824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:41881)
0808 02:28:58.836048 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:58.836165 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 02:28:58.837392 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:58.836897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:51119)
0808 02:28:58.837095 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:58.837213 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 02:28:58.837687 5732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 02:28:59.185678 5735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:28:59.186012 5735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:28:59.186203 5735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:28:59.193503 5735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:28:59.193665 5735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:28:59.193852 5735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:28:59.203555 5735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:42377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=38611
--tserver_master_addrs=127.19.231.62:43731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:28:59.204962 5735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:28:59.206351 5735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:28:59.220084 5740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:59.220148 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:28:59.220039 5743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:28:59.220959 5735 server_base.cc:1034] running on GCE node
I20260808 02:28:59.221582 5735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:28:59.222564 5735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:28:59.223831 5735 hybrid_clock.cc:648] HybridClock initialized: now 1786156139223784 us; error 33 us; skew 500 ppm
I20260808 02:28:59.224190 5735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:28:59 dist-test-slave-pgkr krb5kdc[4146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156139, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:28:59.232779 5735 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:28:59.234473 5735 webserver.cc:460] Webserver started at http://127.19.231.3:38611/ using document root <none> and password file <none>
I20260808 02:28:59.235273 5735 fs_manager.cc:359] Metadata directory not provided
I20260808 02:28:59.235443 5735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:28:59.241504 5735 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 02:28:59.244608 5750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:59.246737 5735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 02:28:59.246961 5735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "c39eebd671cc45559fa6705a71c27222"
format_stamp: "Formatted at 2026-08-08 02:28:13 on dist-test-slave-pgkr"
server_key: "4452f183100bbffd3528d8dfc218066c"
server_key_iv: "6cea29e13644d8f9aa47634e3fb8b7df"
server_key_version: "encryptionkey@0"
I20260808 02:28:59.248675 5735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:28:59.284236 5735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:28:59.289091 5735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:28:59.289388 5735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:28:59.290462 5735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:28:59.291980 5735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:28:59.292109 5735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:59.292268 5735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:28:59.292363 5735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:28:59.327940 5735 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:42377
I20260808 02:28:59.328027 5871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:42377 every 8 connection(s)
I20260808 02:28:59.329470 5735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:28:59.334345 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:59.332522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:49269)
0808 02:28:59.333244 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:59.333494 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":285,"threads_started":1}
W20260808 02:28:59.335139 5872 heartbeater.cc:644] Failed to heartbeat to 127.19.231.62:43731 (0 consecutive failures): Network error: Failed to ping master at 127.19.231.62:43731: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
I20260808 02:28:59.335987 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:59.335645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:59195)
0808 02:28:59.335804 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:59.335878 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 02:28:59.336872 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:59.336503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:43841)
0808 02:28:59.336680 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:59.336759 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260808 02:28:59.337116 5872 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 02:28:59.339203 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 5735
I20260808 02:28:59.343485 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:59.342702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:46335)
0808 02:28:59.343142 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:59.343250 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":201,"threads_started":1}
I20260808 02:28:59.839197 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:28:59.838313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:38019)
0808 02:28:59.838823 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 02:28:59.838934 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":219,"threads_started":1}
I20260808 02:29:00.338348 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:00.337626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:33087)
0808 02:29:00.338040 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:00.338123 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":205,"threads_started":1}
I20260808 02:29:00.345192 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:00.344412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:60813)
0808 02:29:00.344934 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:00.345016 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":284,"threads_started":1}
I20260808 02:29:00.840975 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:00.840216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:56757)
0808 02:29:00.840684 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:00.840769 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":232,"threads_started":1}
I20260808 02:29:01.340034 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:01.339239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:36237)
0808 02:29:01.339751 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:01.339833 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":278,"threads_started":1}
I20260808 02:29:01.346940 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:01.346036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:51281)
0808 02:29:01.346652 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:01.346758 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":346,"threads_started":1}
I20260808 02:29:01.842548 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:01.841826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:46151)
0808 02:29:01.842283 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:01.842367 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":235,"threads_started":1}
I20260808 02:29:02.341658 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:02.340884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:48423)
0808 02:29:02.341408 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:02.341491 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":321,"threads_started":1}
I20260808 02:29:02.348471 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:02.347753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:40549)
0808 02:29:02.348219 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:02.348300 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":232,"threads_started":1}
I20260808 02:29:02.844394 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:02.843543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:50389)
0808 02:29:02.844072 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:02.844175 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
I20260808 02:29:03.343463 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:03.342735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:55345)
0808 02:29:03.343196 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:03.343290 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20260808 02:29:03.350468 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:03.349483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:44359)
0808 02:29:03.350167 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:03.350263 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":400,"threads_started":1}
I20260808 02:29:03.846493 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:03.845372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:37697)
0808 02:29:03.846027 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:03.846185 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":298,"threads_started":1}
I20260808 02:29:04.345216 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:04.344462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:36709)
0808 02:29:04.344958 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:04.345040 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260808 02:29:04.352437 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:04.351435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:49605)
0808 02:29:04.352041 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:04.352173 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":326,"threads_started":1}
I20260808 02:29:04.848409 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:04.847518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:34961)
0808 02:29:04.848091 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:04.848221 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":313,"threads_started":1}
I20260808 02:29:05.346920 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:05.346182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:49451)
0808 02:29:05.346650 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:05.346756 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260808 02:29:05.354410 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:05.353426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:58903)
0808 02:29:05.354022 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:05.354114 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":306,"threads_started":1}
I20260808 02:29:05.850338 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:05.849487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:33035)
0808 02:29:05.850011 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:05.850096 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":234,"threads_started":1}
I20260808 02:29:06.348628 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:06.347827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:58063)
0808 02:29:06.348335 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:06.348422 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":214,"threads_started":1}
I20260808 02:29:06.356190 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:06.355340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:60783)
0808 02:29:06.355898 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:06.356001 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":274,"threads_started":1}
I20260808 02:29:06.852013 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:06.851318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:46047)
0808 02:29:06.851735 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:06.851819 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260808 02:29:07.350325 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:07.349611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:44117)
0808 02:29:07.350061 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:07.350143 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260808 02:29:07.357968 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:07.357089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:51229)
0808 02:29:07.357695 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:07.357778 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260808 02:29:07.853895 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:07.853113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:52749)
0808 02:29:07.853571 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:07.853657 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":228,"threads_started":1}
I20260808 02:29:08.352123 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:08.351327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:58599)
0808 02:29:08.351839 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:08.351952 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":261,"threads_started":1}
I20260808 02:29:08.359594 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:08.358902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:57859)
0808 02:29:08.359354 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:08.359439 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":251,"threads_started":1}
I20260808 02:29:08.855892 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:08.854975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:50611)
0808 02:29:08.855511 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:08.855627 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
I20260808 02:29:09.354046 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:09.353148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:49353)
0808 02:29:09.353764 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:09.353850 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":384,"threads_started":1}
I20260808 02:29:09.361120 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:09.360438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:48559)
0808 02:29:09.360851 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:09.360948 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260808 02:29:09.857546 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:09.856744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:52315)
0808 02:29:09.857267 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:09.857359 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":256,"threads_started":1}
I20260808 02:29:10.355822 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:10.355041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:46557)
0808 02:29:10.355546 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:10.355637 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260808 02:29:10.362777 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:10.361998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:58351)
0808 02:29:10.362467 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:10.362550 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260808 02:29:10.859378 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:10.858676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:50747)
0808 02:29:10.859132 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:10.859214 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260808 02:29:11.357558 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:11.356806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:40219)
0808 02:29:11.357267 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:11.357398 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
I20260808 02:29:11.364547 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:11.363683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:39721)
0808 02:29:11.364222 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:11.364329 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":254,"threads_started":1}
I20260808 02:29:11.861253 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:11.860389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:40157)
0808 02:29:11.860862 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:11.861043 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
I20260808 02:29:12.359547 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:12.358665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:45309)
0808 02:29:12.359216 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:12.359328 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":246,"threads_started":1}
I20260808 02:29:12.366102 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:12.365409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:47077)
0808 02:29:12.365849 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:12.365928 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":221,"threads_started":1}
I20260808 02:29:12.863152 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:12.862255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:34283)
0808 02:29:12.862838 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:12.862938 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":388,"threads_started":1}
I20260808 02:29:13.361171 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:13.360387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:54569)
0808 02:29:13.360843 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:13.360923 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260808 02:29:13.368052 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:13.367258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:39365)
0808 02:29:13.367766 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:13.367875 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":280,"threads_started":1}
I20260808 02:29:13.865211 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:13.864437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.2:52315)
0808 02:29:13.864882 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:13.864982 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260808 02:29:14.346351 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 5409
I20260808 02:29:14.363181 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:14.362355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.3:51327)
0808 02:29:14.362902 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:14.362992 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":316,"threads_started":1}
I20260808 02:29:14.367693 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:14.347595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.19.231.1:43323)
0808 02:29:14.348353 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:14.348369 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:14.348439 (+ 70us) server_negotiation.cc:407] Connection header received
0808 02:29:14.350384 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:14.350394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:14.350920 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:14.351306 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:14.352950 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:14.354315 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:14.355743 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:14.356434 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:14.356637 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:14.360260 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:14.360270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:14.360286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:14.360335 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:14.364218 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:14.364972 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:14.364980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:14.364986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:14.365189 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:14.365595 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:14.365600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:14.365603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:14.366354 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:14.366708 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:14.367019 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 02:29:14.367370 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":315,"threads_started":1}
I20260808 02:29:14.369249 5585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:14.455772 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:14.369004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43731 (local address 127.19.231.1:51081)
0808 02:29:14.455142 (+ 86138us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:14.455405 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.62:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85974,"thread_start_us":246,"threads_started":1}
I20260808 02:29:14.456589 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 5409
I20260808 02:29:14.484741 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 5594
I20260808 02:29:14.503342 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:14.485680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.19.231.2:35445)
0808 02:29:14.486268 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:14.486282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:14.486322 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:29:14.488010 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:14.488021 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:14.488488 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:14.488874 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:14.490391 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:14.491730 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:14.493052 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:14.493743 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:14.493953 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:14.496734 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:14.496743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:14.496759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:14.496799 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:14.499847 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:14.500605 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:14.500612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:14.500619 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:14.500803 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:14.501217 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:14.501221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:14.501223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:14.501758 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:14.502091 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:14.502578 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 02:29:14.503039 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":249,"threads_started":1}
I20260808 02:29:14.504877 5727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:14.594376 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 5594
I20260808 02:29:14.618512 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 5735
I20260808 02:29:14.637307 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:14.619651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.19.231.3:42377)
0808 02:29:14.620297 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:14.620312 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:14.620357 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:29:14.622068 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:14.622079 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:14.622519 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:14.623107 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:14.624704 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:14.625953 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:14.627322 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:14.627991 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:14.628221 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:14.630974 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:14.630982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:14.630997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:14.631036 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:14.633902 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:14.634596 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:14.634603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:14.634653 (+ 50us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:14.634847 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:14.635262 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:14.635268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:14.635272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:14.635820 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:14.636176 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:14.636651 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 02:29:14.636987 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":306,"threads_started":1}
I20260808 02:29:14.638772 5867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:14.726006 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 5735
2026-08-08T02:29:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65401 ms)
[----------] 30 tests from SecurityITest (314776 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:14 dist-test-slave-pgkr krb5kdc[5937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:14 dist-test-slave-pgkr krb5kdc[5937](info): set up 2 sockets
Aug 08 02:29:14 dist-test-slave-pgkr krb5kdc[5937](info): commencing operation
krb5kdc: starting...
W20260808 02:29:15.923671 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:16 dist-test-slave-pgkr krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:29:16.259481 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:39773
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:16.620168 5951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:16.620519 5951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:16.620678 5951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:16.628770 5951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:16.628924 5951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:16.629029 5951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:16.629146 5951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:16.629228 5951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:16.639887 5951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:16.641317 5951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:16.642762 5951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:16.655289 5956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:16.655334 5957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:16.655369 5960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:16.656265 5951 server_base.cc:1034] running on GCE node
I20260808 02:29:16.657051 5951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:16.658468 5951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:16.659700 5951 hybrid_clock.cc:648] HybridClock initialized: now 1786156156659638 us; error 59 us; skew 500 ppm
I20260808 02:29:16.660070 5951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:16 dist-test-slave-pgkr krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156156, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:16.663373 5951 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:29:16.664927 5951 webserver.cc:460] Webserver started at http://127.19.231.62:35431/ using document root <none> and password file <none>
I20260808 02:29:16.665678 5951 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:16.665817 5951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:16.666141 5951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:16.669301 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "8590dec8041e46e0b9dc9a409324b0df"
format_stamp: "Formatted at 2026-08-08 02:29:16 on dist-test-slave-pgkr"
server_key: "5315715bfc5c0ff7577950b264b1556b"
server_key_iv: "3e3662dea4b66d7eb0cc2ecde3c7e1cb"
server_key_version: "encryptionkey@0"
I20260808 02:29:16.670099 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "8590dec8041e46e0b9dc9a409324b0df"
format_stamp: "Formatted at 2026-08-08 02:29:16 on dist-test-slave-pgkr"
server_key: "5315715bfc5c0ff7577950b264b1556b"
server_key_iv: "3e3662dea4b66d7eb0cc2ecde3c7e1cb"
server_key_version: "encryptionkey@0"
I20260808 02:29:16.675468 5951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 02:29:16.679090 5968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:16.680655 5951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:29:16.680874 5951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "8590dec8041e46e0b9dc9a409324b0df"
format_stamp: "Formatted at 2026-08-08 02:29:16 on dist-test-slave-pgkr"
server_key: "5315715bfc5c0ff7577950b264b1556b"
server_key_iv: "3e3662dea4b66d7eb0cc2ecde3c7e1cb"
server_key_version: "encryptionkey@0"
I20260808 02:29:16.681116 5951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:16.701177 5951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:16.705879 5951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:16.706197 5951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:16.726030 5951 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:39773
I20260808 02:29:16.726055 6024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:39773 every 8 connection(s)
I20260808 02:29:16.727499 5951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:16.731423 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:16.732730 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 5951
I20260808 02:29:16.732977 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:16.733667 20380 external_mini_cluster.cc:1428] Setting key 793f5b71d67625dd7d537a984e9b7f41
I20260808 02:29:16.746392 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df: Bootstrap starting.
Aug 08 02:29:16 dist-test-slave-pgkr krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:16.749635 6026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:16.751277 6026 log.cc:824] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:16.754918 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df: No bootstrap required, opened a new log
I20260808 02:29:16.759042 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:16.735752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.19.231.62:39773)
0808 02:29:16.736725 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:16.736742 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:16.736794 (+ 52us) server_negotiation.cc:407] Connection header received
0808 02:29:16.738573 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:16.738602 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:16.739184 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:16.739647 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:16.741029 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:16.743169 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:16.744658 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:16.745719 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:16.745899 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:16.750051 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:16.750065 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:16.750086 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:16.750152 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:16.754245 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:16.755050 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:16.755058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:16.755066 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:16.755291 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:16.755679 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:16.755687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:16.755690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:16.756410 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:16.756713 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:16.757133 (+ 420us) server_negotiation.cc:299] Negotiation successful
0808 02:29:16.757609 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":468,"threads_started":1}
I20260808 02:29:16.764735 6026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8590dec8041e46e0b9dc9a409324b0df" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39773 } }
I20260808 02:29:16.765066 6026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:16.765141 6026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8590dec8041e46e0b9dc9a409324b0df, State: Initialized, Role: FOLLOWER
I20260808 02:29:16.765614 6026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8590dec8041e46e0b9dc9a409324b0df" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39773 } }
I20260808 02:29:16.765825 6026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:16.765923 6026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:16.766059 6026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:16.768543 6026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8590dec8041e46e0b9dc9a409324b0df" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39773 } }
I20260808 02:29:16.769011 6026 leader_election.cc:302] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8590dec8041e46e0b9dc9a409324b0df; no voters:
I20260808 02:29:16.769567 6026 leader_election.cc:288] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:16.769790 6031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:16.770697 6031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [term 1 LEADER]: Becoming Leader. State: Replica: 8590dec8041e46e0b9dc9a409324b0df, State: Running, Role: LEADER
I20260808 02:29:16.771231 6031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8590dec8041e46e0b9dc9a409324b0df" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39773 } }
I20260808 02:29:16.771816 6026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:16.774370 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8590dec8041e46e0b9dc9a409324b0df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8590dec8041e46e0b9dc9a409324b0df" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39773 } } }
I20260808 02:29:16.774911 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:16.774816 6033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8590dec8041e46e0b9dc9a409324b0df. Latest consensus state: current_term: 1 leader_uuid: "8590dec8041e46e0b9dc9a409324b0df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8590dec8041e46e0b9dc9a409324b0df" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 39773 } } }
I20260808 02:29:16.775199 6033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:16.777074 6041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:16.782280 6041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:16.789126 6049 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:29:16.792236 6041 catalog_manager.cc:1347] Generated new cluster ID: 359b6be3b6114be3b9ebde9ffc95286b
I20260808 02:29:16.792392 6041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:16.811764 6041 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:16.813205 6041 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:16.819525 6041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df: Generated new TSK 0
I20260808 02:29:16.820313 6041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:29:16.898746 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39773
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:17.261305 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:17.261588 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:17.261685 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:17.269001 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:17.269114 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:17.269243 6054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:29:17.279693 6054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:17.280988 6054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:17.282281 6054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:17.296134 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:17.296221 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:17.296187 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:17.297060 6054 server_base.cc:1034] running on GCE node
I20260808 02:29:17.297732 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:17.298740 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:17.299978 6054 hybrid_clock.cc:648] HybridClock initialized: now 1786156157299931 us; error 55 us; skew 500 ppm
I20260808 02:29:17.300328 6054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:17 dist-test-slave-pgkr krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156157, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:17.303507 6054 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:29:17.304987 6054 webserver.cc:460] Webserver started at http://127.19.231.1:45615/ using document root <none> and password file <none>
I20260808 02:29:17.305675 6054 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:17.305826 6054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:17.306172 6054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:17.309170 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "8a104fe8eae54262a49b314e65d012fd"
format_stamp: "Formatted at 2026-08-08 02:29:17 on dist-test-slave-pgkr"
server_key: "c5c3dbf3d4d991553509d6990978c983"
server_key_iv: "cfe945d2b4fce9da9806c9f71282b720"
server_key_version: "encryptionkey@0"
I20260808 02:29:17.309926 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "8a104fe8eae54262a49b314e65d012fd"
format_stamp: "Formatted at 2026-08-08 02:29:17 on dist-test-slave-pgkr"
server_key: "c5c3dbf3d4d991553509d6990978c983"
server_key_iv: "cfe945d2b4fce9da9806c9f71282b720"
server_key_version: "encryptionkey@0"
I20260808 02:29:17.315109 6054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260808 02:29:17.318471 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:17.320106 6054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:17.320298 6054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "8a104fe8eae54262a49b314e65d012fd"
format_stamp: "Formatted at 2026-08-08 02:29:17 on dist-test-slave-pgkr"
server_key: "c5c3dbf3d4d991553509d6990978c983"
server_key_iv: "cfe945d2b4fce9da9806c9f71282b720"
server_key_version: "encryptionkey@0"
I20260808 02:29:17.320521 6054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:17.346312 6054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:17.351446 6054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:17.351790 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:17.352893 6054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:17.354403 6054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:17.354496 6054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:17.354712 6054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:17.354776 6054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:17.392414 6054 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:39989
I20260808 02:29:17.392462 6189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:39989 every 8 connection(s)
I20260808 02:29:17.393841 6054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:29:17.402462 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 6054
I20260808 02:29:17.402772 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:29:17.403479 20380 external_mini_cluster.cc:1428] Setting key efe9f1d9fef3bb7f1f23fcb32352e3a9
Aug 08 02:29:17 dist-test-slave-pgkr krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156157, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:17.416491 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:17.396485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:35053 (local address 127.19.231.62:39773)
0808 02:29:17.397018 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:17.397023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:17.397904 (+ 881us) server_negotiation.cc:407] Connection header received
0808 02:29:17.399871 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:17.399877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:17.399976 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:17.400098 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:17.401619 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.402383 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.404111 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.404522 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.404667 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:17.408471 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:17.408480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:17.408488 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:17.408538 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:17.412514 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:17.413461 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:17.413470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:17.413476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:17.413658 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:17.414164 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:17.414172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:17.414178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:17.414607 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:17.414798 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:17.415671 (+ 873us) server_negotiation.cc:299] Negotiation successful
0808 02:29:17.415947 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":315,"threads_started":1}
I20260808 02:29:17.417476 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:17.397004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39773 (local address 127.19.231.1:35053)
0808 02:29:17.397745 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:17.397785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:17.399591 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:17.400352 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:17.400360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:17.400871 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:17.401442 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:17.401459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.402550 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.402554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:17.403960 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:17.403973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.404775 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.404785 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:17.405035 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:17.405755 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:17.405776 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:17.408190 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:17.412682 (+ 4492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:17.412693 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:17.412711 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:17.413298 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:17.413782 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:17.413788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:17.413794 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:17.414030 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:17.414767 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:17.414777 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:17.415205 (+ 428us) client_negotiation.cc:769] Sending connection context
0808 02:29:17.415477 (+ 272us) client_negotiation.cc:240] Negotiation successful
0808 02:29:17.416012 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":287,"threads_started":1}
I20260808 02:29:17.419287 6190 heartbeater.cc:347] Connected to a master server at 127.19.231.62:39773
I20260808 02:29:17.419595 6190 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:17.420454 6190 heartbeater.cc:511] Master 127.19.231.62:39773 requested a full tablet report, sending...
I20260808 02:29:17.422856 5986 ts_manager.cc:194] Registered new tserver with Master: 8a104fe8eae54262a49b314e65d012fd (127.19.231.1:39989)
I20260808 02:29:17.424408 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:35053
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:29:17.477598 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39773
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:17.839048 6196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:17.839406 6196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:17.839620 6196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:17.847436 6196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:17.847575 6196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:17.847769 6196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:29:17.858122 6196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:17.859661 6196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:17.861158 6196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:17.876264 6205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:17.876209 6203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:17.876292 6202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:17.877193 6196 server_base.cc:1034] running on GCE node
I20260808 02:29:17.877832 6196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:17.879025 6196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:17.880308 6196 hybrid_clock.cc:648] HybridClock initialized: now 1786156157880197 us; error 89 us; skew 500 ppm
I20260808 02:29:17.880676 6196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:17 dist-test-slave-pgkr krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156157, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:17.884085 6196 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:29:17.885820 6196 webserver.cc:460] Webserver started at http://127.19.231.2:44573/ using document root <none> and password file <none>
I20260808 02:29:17.886552 6196 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:17.886731 6196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:17.887077 6196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:17.890218 6196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "9d978f43f82a496fa95c6273bc2779a5"
format_stamp: "Formatted at 2026-08-08 02:29:17 on dist-test-slave-pgkr"
server_key: "734ca70e192f3f732d1a8eed60a8152d"
server_key_iv: "820e3804e446ae6149370793d5927bb8"
server_key_version: "encryptionkey@0"
I20260808 02:29:17.891063 6196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "9d978f43f82a496fa95c6273bc2779a5"
format_stamp: "Formatted at 2026-08-08 02:29:17 on dist-test-slave-pgkr"
server_key: "734ca70e192f3f732d1a8eed60a8152d"
server_key_iv: "820e3804e446ae6149370793d5927bb8"
server_key_version: "encryptionkey@0"
I20260808 02:29:17.896631 6196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:29:17.900353 6212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:17.901732 6196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:29:17.901954 6196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "9d978f43f82a496fa95c6273bc2779a5"
format_stamp: "Formatted at 2026-08-08 02:29:17 on dist-test-slave-pgkr"
server_key: "734ca70e192f3f732d1a8eed60a8152d"
server_key_iv: "820e3804e446ae6149370793d5927bb8"
server_key_version: "encryptionkey@0"
I20260808 02:29:17.902184 6196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:17.923081 6196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:17.927479 6196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:17.927796 6196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:17.928826 6196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:17.930303 6196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:17.930413 6196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:17.930565 6196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:17.930668 6196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:17.965915 6196 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:41825
I20260808 02:29:17.965977 6335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:41825 every 8 connection(s)
I20260808 02:29:17.967396 6196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:29:17.971052 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 6196
I20260808 02:29:17.971268 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:29:17.971966 20380 external_mini_cluster.cc:1428] Setting key 59668d24330515590730a4c74a823f07
Aug 08 02:29:17 dist-test-slave-pgkr krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156157, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:17.991918 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:17.970223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:36753 (local address 127.19.231.62:39773)
0808 02:29:17.970761 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:17.970770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:17.972021 (+ 1251us) server_negotiation.cc:407] Connection header received
0808 02:29:17.974326 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:17.974332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:17.974425 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:17.974520 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:17.976862 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.977696 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.980245 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.980758 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.980868 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:17.985619 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:17.985625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:17.985628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:17.985665 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:17.988488 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:17.989328 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:17.989332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:17.989334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:17.989472 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:17.989952 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:17.989956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:17.989958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:17.990240 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:17.990403 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:17.991243 (+ 840us) server_negotiation.cc:299] Negotiation successful
0808 02:29:17.991490 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
I20260808 02:29:17.992990 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:17.970870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39773 (local address 127.19.231.2:36753)
0808 02:29:17.971822 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:17.971882 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:17.974016 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:17.974862 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:17.974875 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:17.975725 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:17.976638 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:17.976668 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.977871 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.977879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:17.980076 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:17.980095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:17.981014 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:17.981024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:17.981383 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:17.982174 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:17.982227 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:17.985279 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:17.988661 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:17.988671 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:17.988689 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:17.989177 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:17.989595 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:17.989601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:17.989605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:17.989816 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:17.990388 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:17.990396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:17.990857 (+ 461us) client_negotiation.cc:769] Sending connection context
0808 02:29:17.991144 (+ 287us) client_negotiation.cc:240] Negotiation successful
0808 02:29:17.991724 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":380,"threads_started":1}
I20260808 02:29:17.994601 6336 heartbeater.cc:347] Connected to a master server at 127.19.231.62:39773
I20260808 02:29:17.994977 6336 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:17.995983 6336 heartbeater.cc:511] Master 127.19.231.62:39773 requested a full tablet report, sending...
I20260808 02:29:17.997999 5986 ts_manager.cc:194] Registered new tserver with Master: 9d978f43f82a496fa95c6273bc2779a5 (127.19.231.2:41825)
I20260808 02:29:17.999395 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:36753
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:29:18.048465 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39773
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:18.416149 6344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:18.416460 6344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:18.416567 6344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:18.424077 6344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:18.424260 6344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:18.424443 6344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:29:18.428283 6190 heartbeater.cc:503] Master 127.19.231.62:39773 was elected leader, sending a full tablet report...
I20260808 02:29:18.435022 6344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:39773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:18.436375 6344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:18.437711 6344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:18.451706 6349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:18.451704 6350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:18.451892 6352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:18.453033 6344 server_base.cc:1034] running on GCE node
I20260808 02:29:18.453660 6344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:18.454762 6344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:18.456023 6344 hybrid_clock.cc:648] HybridClock initialized: now 1786156158455957 us; error 63 us; skew 500 ppm
I20260808 02:29:18.456391 6344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:18 dist-test-slave-pgkr krb5kdc[5937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:18.459594 6344 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:29:18.461062 6344 webserver.cc:460] Webserver started at http://127.19.231.3:41613/ using document root <none> and password file <none>
I20260808 02:29:18.461755 6344 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:18.461907 6344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:18.462245 6344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:18.465221 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "228c2e0a56de4fa890394ceab4cd70a0"
format_stamp: "Formatted at 2026-08-08 02:29:18 on dist-test-slave-pgkr"
server_key: "b7177e4425fc858958ad90562b14d798"
server_key_iv: "9deae0621b2b9f1fc0ba778bb6d41ed6"
server_key_version: "encryptionkey@0"
I20260808 02:29:18.465986 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "228c2e0a56de4fa890394ceab4cd70a0"
format_stamp: "Formatted at 2026-08-08 02:29:18 on dist-test-slave-pgkr"
server_key: "b7177e4425fc858958ad90562b14d798"
server_key_iv: "9deae0621b2b9f1fc0ba778bb6d41ed6"
server_key_version: "encryptionkey@0"
I20260808 02:29:18.471230 6344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:29:18.474959 6359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:18.476502 6344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:29:18.476680 6344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "228c2e0a56de4fa890394ceab4cd70a0"
format_stamp: "Formatted at 2026-08-08 02:29:18 on dist-test-slave-pgkr"
server_key: "b7177e4425fc858958ad90562b14d798"
server_key_iv: "9deae0621b2b9f1fc0ba778bb6d41ed6"
server_key_version: "encryptionkey@0"
I20260808 02:29:18.476886 6344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:18.497021 6344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:18.501484 6344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:18.501813 6344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:18.502887 6344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:18.504268 6344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:18.504372 6344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:18.504528 6344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:18.504602 6344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:18.540038 6344 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:32967
I20260808 02:29:18.540130 6482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:32967 every 8 connection(s)
I20260808 02:29:18.541376 6344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:29:18.543740 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 6344
I20260808 02:29:18.543972 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:29:18.544580 20380 external_mini_cluster.cc:1428] Setting key 9d3d546e0fd6afa37287ba7c013efdb2
Aug 08 02:29:18 dist-test-slave-pgkr krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:18.563681 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.544401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:42841 (local address 127.19.231.62:39773)
0808 02:29:18.544946 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.544957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.546085 (+ 1128us) server_negotiation.cc:407] Connection header received
0808 02:29:18.548664 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.548670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.548799 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.548932 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:18.550686 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.551520 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.553172 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.553605 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.553756 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.557400 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:18.557405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:18.557409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:18.557443 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:18.560777 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:18.561493 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:18.561497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:18.561500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:18.561689 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:18.562089 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:18.562093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:18.562095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:18.562377 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:18.562547 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.563196 (+ 649us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.563357 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":231,"threads_started":1}
I20260808 02:29:18.564599 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.544954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:39773 (local address 127.19.231.3:42841)
0808 02:29:18.545876 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:18.545927 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:18.548371 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:18.549266 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:18.549279 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:18.549861 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:18.550468 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.550487 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.551704 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.551709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.553000 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.553023 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.553864 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.553872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.554133 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.554954 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:18.554977 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:18.557184 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:18.560897 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:18.560904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:18.560915 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:18.561352 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:18.561799 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:18.561802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:18.561804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:18.561984 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:18.562487 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:18.562493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:18.562853 (+ 360us) client_negotiation.cc:769] Sending connection context
0808 02:29:18.563088 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 02:29:18.563452 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":378,"threads_started":1}
I20260808 02:29:18.566004 6483 heartbeater.cc:347] Connected to a master server at 127.19.231.62:39773
I20260808 02:29:18.566336 6483 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:18.567107 6483 heartbeater.cc:511] Master 127.19.231.62:39773 requested a full tablet report, sending...
I20260808 02:29:18.568504 5986 ts_manager.cc:194] Registered new tserver with Master: 228c2e0a56de4fa890394ceab4cd70a0 (127.19.231.3:32967)
I20260808 02:29:18.569250 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:42841
I20260808 02:29:18.572820 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:29:18.591617 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.577874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.19.231.62:39773)
0808 02:29:18.578060 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.578068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.578511 (+ 443us) server_negotiation.cc:407] Connection header received
0808 02:29:18.578760 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.578765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.578834 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.578939 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:18.580462 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.581245 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.582694 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.583123 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.583297 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.585957 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:18.585963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:18.585974 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:18.586006 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:18.588723 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:18.589477 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:18.589483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:18.589487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:18.589614 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:18.590045 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:18.590050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:18.590053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:18.590386 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:18.590685 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.591051 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.591280 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:29:18.595892 5986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:29:18.602924 5986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:29:18.627589 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.615252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:39989 (local address 127.0.0.1:42058)
0808 02:29:18.618206 (+ 2954us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:18.618230 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:18.618370 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:18.620391 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:18.620397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:18.620430 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:18.620875 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.620890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.622490 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.622495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.624891 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.624904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.626576 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.626582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.626812 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.626874 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 02:29:18.626995 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 02:29:18.627201 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784,"thread_start_us":1387,"threads_started":1}
I20260808 02:29:18.628615 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.615886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.19.231.1:39989)
0808 02:29:18.618963 (+ 3077us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.618975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.619013 (+ 38us) server_negotiation.cc:407] Connection header received
0808 02:29:18.619149 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.619161 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.620157 (+ 996us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.620352 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:18.621139 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.622334 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.625133 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.626367 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.626518 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.627960 (+ 1442us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.628045 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.628255 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2878,"thread_start_us":2720,"threads_started":1}
I20260808 02:29:18.628829 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.614932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:32967 (local address 127.0.0.1:46918)
0808 02:29:18.616040 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:18.616096 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:18.616329 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:18.620471 (+ 4142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:18.620480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:18.620530 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:18.621198 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.621216 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.623350 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.623355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.625936 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.625952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.627881 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.627888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.628079 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.628117 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:29:18.628206 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 02:29:18.628343 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":107,"spinlock_wait_cycles":1920,"thread_start_us":4877,"threads_started":1}
I20260808 02:29:18.629384 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.615268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.19.231.3:32967)
0808 02:29:18.619874 (+ 4606us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.619882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.619913 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:29:18.620010 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.620017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.620258 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.621171 (+ 913us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:18.621506 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.623171 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.626135 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.627648 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.628429 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.628666 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.628870 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.629057 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4416,"thread_start_us":4201,"threads_started":1}
I20260808 02:29:18.629458 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.617806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.19.231.2:41825)
0808 02:29:18.618381 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.618393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.620200 (+ 1807us) server_negotiation.cc:407] Connection header received
0808 02:29:18.620354 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.620364 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.620587 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.620802 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:18.621558 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.623193 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.626135 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.627648 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.627786 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.628004 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.628929 (+ 925us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.629113 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":285,"threads_started":1}
I20260808 02:29:18.629496 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.617503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41825 (local address 127.0.0.1:54378)
0808 02:29:18.620060 (+ 2557us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:18.620077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:18.620215 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:18.620819 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:18.620823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:18.620852 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:18.621391 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.621406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.623352 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.623356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.625941 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.625951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.628203 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.628209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.628365 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.628394 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:29:18.628488 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 02:29:18.628612 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2445,"mutex_wait_us":106,"thread_start_us":334,"threads_started":1}
I20260808 02:29:18.633324 6120 tablet_service.cc:1467] Processing CreateTablet for tablet 330fba7b57ed403e8c6a76c25a4e7826 (DEFAULT_TABLE table=test-table [id=4727212175b545c6bf6098baa0a05636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:18.633610 6413 tablet_service.cc:1467] Processing CreateTablet for tablet 330fba7b57ed403e8c6a76c25a4e7826 (DEFAULT_TABLE table=test-table [id=4727212175b545c6bf6098baa0a05636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:18.633762 6267 tablet_service.cc:1467] Processing CreateTablet for tablet 330fba7b57ed403e8c6a76c25a4e7826 (DEFAULT_TABLE table=test-table [id=4727212175b545c6bf6098baa0a05636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:18.635037 6267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330fba7b57ed403e8c6a76c25a4e7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:18.635066 6120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330fba7b57ed403e8c6a76c25a4e7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:18.635067 6413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330fba7b57ed403e8c6a76c25a4e7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:18.647929 6502 tablet_bootstrap.cc:492] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Bootstrap starting.
I20260808 02:29:18.650018 6504 tablet_bootstrap.cc:492] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Bootstrap starting.
I20260808 02:29:18.650859 6506 tablet_bootstrap.cc:492] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Bootstrap starting.
I20260808 02:29:18.651966 6502 tablet_bootstrap.cc:654] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:18.653358 6502 log.cc:824] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:18.654793 6506 tablet_bootstrap.cc:654] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:18.656193 6506 log.cc:824] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:18.656581 6502 tablet_bootstrap.cc:492] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: No bootstrap required, opened a new log
I20260808 02:29:18.656924 6502 ts_tablet_manager.cc:1397] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 02:29:18.658324 6504 tablet_bootstrap.cc:654] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:18.659507 6506 tablet_bootstrap.cc:492] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: No bootstrap required, opened a new log
I20260808 02:29:18.659618 6504 log.cc:824] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:18.659806 6506 ts_tablet_manager.cc:1397] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260808 02:29:18.662839 6504 tablet_bootstrap.cc:492] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: No bootstrap required, opened a new log
I20260808 02:29:18.663115 6504 ts_tablet_manager.cc:1397] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260808 02:29:18.665062 6502 raft_consensus.cc:348] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.665460 6502 raft_consensus.cc:374] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:18.665596 6502 raft_consensus.cc:729] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d978f43f82a496fa95c6273bc2779a5, State: Initialized, Role: FOLLOWER
I20260808 02:29:18.666268 6502 consensus_queue.cc:260] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.667958 6336 heartbeater.cc:503] Master 127.19.231.62:39773 was elected leader, sending a full tablet report...
I20260808 02:29:18.668696 6502 ts_tablet_manager.cc:1428] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260808 02:29:18.670463 6506 raft_consensus.cc:348] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.670956 6506 raft_consensus.cc:374] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:18.671075 6506 raft_consensus.cc:729] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a104fe8eae54262a49b314e65d012fd, State: Initialized, Role: FOLLOWER
I20260808 02:29:18.671707 6506 consensus_queue.cc:260] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.675141 6504 raft_consensus.cc:348] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.675603 6504 raft_consensus.cc:374] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:18.675737 6504 raft_consensus.cc:729] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228c2e0a56de4fa890394ceab4cd70a0, State: Initialized, Role: FOLLOWER
I20260808 02:29:18.676136 6506 ts_tablet_manager.cc:1428] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260808 02:29:18.676419 6504 consensus_queue.cc:260] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.677717 6483 heartbeater.cc:503] Master 127.19.231.62:39773 was elected leader, sending a full tablet report...
I20260808 02:29:18.678239 6504 ts_tablet_manager.cc:1428] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260808 02:29:18.719463 6337 tablet.cc:2369] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:18.720896 6512 raft_consensus.cc:482] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:18.721174 6512 raft_consensus.cc:504] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.722891 6512 leader_election.cc:288] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a104fe8eae54262a49b314e65d012fd (127.19.231.1:39989), 9d978f43f82a496fa95c6273bc2779a5 (127.19.231.2:41825)
I20260808 02:29:18.729687 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.723278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:39989 (local address 127.19.231.3:59121)
0808 02:29:18.723514 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:18.723535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:18.723662 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:18.724105 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:18.724114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:18.724166 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:18.724477 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.724488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.725709 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.725716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.727359 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.727375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.728932 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.728938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.729130 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.729165 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:29:18.729266 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 02:29:18.729399 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:29:18.729712 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.723350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:59121 (local address 127.19.231.1:39989)
0808 02:29:18.723551 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.723556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.723581 (+ 25us) server_negotiation.cc:407] Connection header received
0808 02:29:18.723853 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.723859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.723949 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.724056 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:18.724619 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.725544 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.727534 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.728785 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.728936 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.729008 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.729312 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.729462 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:29:18.731048 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.724033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:43743 (local address 127.19.231.2:41825)
0808 02:29:18.724290 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.724298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.724834 (+ 536us) server_negotiation.cc:407] Connection header received
0808 02:29:18.725050 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.725054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.725118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.725212 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:18.725904 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.727019 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.728769 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.730043 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.730198 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.730273 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.730666 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.730807 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 02:29:18.731107 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.723802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41825 (local address 127.19.231.3:43743)
0808 02:29:18.724735 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:18.724750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:18.724881 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:18.725241 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:18.725245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:18.725271 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:18.725598 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.725644 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.727180 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.727186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.728604 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:18.728624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.730253 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.730259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:18.730459 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.730499 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:29:18.730603 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:29:18.730763 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":568,"threads_started":1}
I20260808 02:29:18.730949 6140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330fba7b57ed403e8c6a76c25a4e7826" candidate_uuid: "228c2e0a56de4fa890394ceab4cd70a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a104fe8eae54262a49b314e65d012fd" is_pre_election: true
I20260808 02:29:18.731596 6140 raft_consensus.cc:2454] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228c2e0a56de4fa890394ceab4cd70a0 in term 0.
I20260808 02:29:18.732272 6362 leader_election.cc:302] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228c2e0a56de4fa890394ceab4cd70a0, 8a104fe8eae54262a49b314e65d012fd; no voters:
I20260808 02:29:18.732282 6288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330fba7b57ed403e8c6a76c25a4e7826" candidate_uuid: "228c2e0a56de4fa890394ceab4cd70a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d978f43f82a496fa95c6273bc2779a5" is_pre_election: true
I20260808 02:29:18.732672 6512 raft_consensus.cc:2781] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:29:18.732824 6512 raft_consensus.cc:482] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:29:18.732788 6288 raft_consensus.cc:2454] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228c2e0a56de4fa890394ceab4cd70a0 in term 0.
I20260808 02:29:18.732940 6512 raft_consensus.cc:3037] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:18.735395 6512 raft_consensus.cc:504] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.735980 6512 leader_election.cc:288] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [CANDIDATE]: Term 1 election: Requested vote from peers 8a104fe8eae54262a49b314e65d012fd (127.19.231.1:39989), 9d978f43f82a496fa95c6273bc2779a5 (127.19.231.2:41825)
I20260808 02:29:18.736536 6288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330fba7b57ed403e8c6a76c25a4e7826" candidate_uuid: "228c2e0a56de4fa890394ceab4cd70a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d978f43f82a496fa95c6273bc2779a5"
I20260808 02:29:18.736538 6140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330fba7b57ed403e8c6a76c25a4e7826" candidate_uuid: "228c2e0a56de4fa890394ceab4cd70a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a104fe8eae54262a49b314e65d012fd"
I20260808 02:29:18.736748 6288 raft_consensus.cc:3037] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:18.736766 6140 raft_consensus.cc:3037] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:18.739148 6288 raft_consensus.cc:2454] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228c2e0a56de4fa890394ceab4cd70a0 in term 1.
I20260808 02:29:18.739681 6362 leader_election.cc:302] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228c2e0a56de4fa890394ceab4cd70a0, 9d978f43f82a496fa95c6273bc2779a5; no voters:
I20260808 02:29:18.740088 6512 raft_consensus.cc:2781] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:18.740366 6140 raft_consensus.cc:2454] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228c2e0a56de4fa890394ceab4cd70a0 in term 1.
I20260808 02:29:18.741312 6512 raft_consensus.cc:686] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 1 LEADER]: Becoming Leader. State: Replica: 228c2e0a56de4fa890394ceab4cd70a0, State: Running, Role: LEADER
I20260808 02:29:18.741833 6512 consensus_queue.cc:237] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } }
I20260808 02:29:18.746408 5984 catalog_manager.cc:5495] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 228c2e0a56de4fa890394ceab4cd70a0 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "228c2e0a56de4fa890394ceab4cd70a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "228c2e0a56de4fa890394ceab4cd70a0" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 32967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 } health_report { overall_health: UNKNOWN } } }
I20260808 02:29:18.784060 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.778502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.19.231.3:32967)
0808 02:29:18.778749 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.778757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.778777 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:29:18.779024 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.779028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.779123 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.779278 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:29:18.779654 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.780446 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.781822 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.782239 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.782376 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.782942 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:29:18.783455 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:29:18.783619 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.783711 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.783791 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:29:18.792367 6288 raft_consensus.cc:1261] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 1 FOLLOWER]: Refusing update from remote peer 228c2e0a56de4fa890394ceab4cd70a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:18.792366 6140 raft_consensus.cc:1261] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 1 FOLLOWER]: Refusing update from remote peer 228c2e0a56de4fa890394ceab4cd70a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 02:29:18.793102 6484 tablet.cc:2369] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:18.793725 6514 consensus_queue.cc:1035] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a104fe8eae54262a49b314e65d012fd" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:18.794181 6512 consensus_queue.cc:1035] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d978f43f82a496fa95c6273bc2779a5" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:18.807437 6518 mvcc.cc:205] Tried to move back new op lower bound from 7316095626402529280 to 7316095626210738176. Current Snapshot: MvccSnapshot[applied={T|T < 7316095626402529280}]
I20260808 02:29:18.822854 5983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:18.823205 5983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:18.827401 5983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8590dec8041e46e0b9dc9a409324b0df: Sending DeleteTablet for 3 replicas of tablet 330fba7b57ed403e8c6a76c25a4e7826
I20260808 02:29:18.828637 6413 tablet_service.cc:1514] Processing DeleteTablet for tablet 330fba7b57ed403e8c6a76c25a4e7826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:18 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:46918
I20260808 02:29:18.828824 6267 tablet_service.cc:1514] Processing DeleteTablet for tablet 330fba7b57ed403e8c6a76c25a4e7826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:18 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:54378
I20260808 02:29:18.828917 6120 tablet_service.cc:1514] Processing DeleteTablet for tablet 330fba7b57ed403e8c6a76c25a4e7826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:18 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:42058
I20260808 02:29:18.831214 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 6054
I20260808 02:29:18.841085 6530 tablet_replica.cc:330] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: stopping tablet replica
I20260808 02:29:18.841876 6530 raft_consensus.cc:2229] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:29:18.842969 6530 raft_consensus.cc:2258] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:18.844522 6531 tablet_replica.cc:330] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: stopping tablet replica
I20260808 02:29:18.845134 6531 raft_consensus.cc:2229] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:18.845361 6532 tablet_replica.cc:330] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: stopping tablet replica
I20260808 02:29:18.845438 6530 ts_tablet_manager.cc:1905] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:18.845826 6531 raft_consensus.cc:2258] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:18.846068 6532 raft_consensus.cc:2229] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:18.846706 6532 raft_consensus.cc:2258] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:18.848153 6531 ts_tablet_manager.cc:1905] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:18.848827 6532 ts_tablet_manager.cc:1905] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:18.854099 6530 ts_tablet_manager.cc:1918] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:18.854449 6530 log.cc:1196] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/330fba7b57ed403e8c6a76c25a4e7826
I20260808 02:29:18.855139 6530 ts_tablet_manager.cc:1939] T 330fba7b57ed403e8c6a76c25a4e7826 P 228c2e0a56de4fa890394ceab4cd70a0: Deleting consensus metadata
I20260808 02:29:18.857142 5969 catalog_manager.cc:4841] TS 228c2e0a56de4fa890394ceab4cd70a0 (127.19.231.3:32967): tablet 330fba7b57ed403e8c6a76c25a4e7826 (table test-table [id=4727212175b545c6bf6098baa0a05636]) successfully deleted
I20260808 02:29:18.857924 6532 ts_tablet_manager.cc:1918] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:18.858109 6532 log.cc:1196] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/330fba7b57ed403e8c6a76c25a4e7826
I20260808 02:29:18.858908 6532 ts_tablet_manager.cc:1939] T 330fba7b57ed403e8c6a76c25a4e7826 P 8a104fe8eae54262a49b314e65d012fd: Deleting consensus metadata
I20260808 02:29:18.861272 5971 catalog_manager.cc:4841] TS 8a104fe8eae54262a49b314e65d012fd (127.19.231.1:39989): tablet 330fba7b57ed403e8c6a76c25a4e7826 (table test-table [id=4727212175b545c6bf6098baa0a05636]) successfully deleted
I20260808 02:29:18.862574 6531 ts_tablet_manager.cc:1918] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:18.862799 6531 log.cc:1196] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/330fba7b57ed403e8c6a76c25a4e7826
I20260808 02:29:18.863328 6531 ts_tablet_manager.cc:1939] T 330fba7b57ed403e8c6a76c25a4e7826 P 9d978f43f82a496fa95c6273bc2779a5: Deleting consensus metadata
Aug 08 02:29:18 dist-test-slave-pgkr krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:29:18.864953 5971 catalog_manager.cc:4841] TS 9d978f43f82a496fa95c6273bc2779a5 (127.19.231.2:41825): tablet 330fba7b57ed403e8c6a76c25a4e7826 (table test-table [id=4727212175b545c6bf6098baa0a05636]) successfully deleted
I20260808 02:29:18.871687 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.832357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.19.231.1:39989)
0808 02:29:18.832602 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.832611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.848966 (+ 16355us) server_negotiation.cc:407] Connection header received
0808 02:29:18.849321 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.849328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.849425 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.849499 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:18.855096 (+ 5597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.856270 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.858242 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.859020 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.859177 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:18.864988 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:18.865004 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:18.865022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:18.865066 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:18.868461 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:18.869350 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:18.869358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:18.869360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:18.869496 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:18.869943 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:18.869946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:18.869949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:18.870339 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:18.870572 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:18.871071 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 02:29:18.871269 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:29:18.872386 6185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:18.965201 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 6054
I20260808 02:29:18.991953 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 6196
Aug 08 02:29:18 dist-test-slave-pgkr krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:29:19.013316 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:18.992828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.19.231.2:41825)
0808 02:29:18.993021 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:18.993027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:18.993167 (+ 140us) server_negotiation.cc:407] Connection header received
0808 02:29:18.993386 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:18.993393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:18.993461 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:18.993570 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:18.994981 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.995802 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.997052 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:18.997464 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:18.997614 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:19.007083 (+ 9469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:19.007105 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:19.007119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:19.007160 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:19.010247 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:19.011094 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:19.011102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:19.011104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:19.011236 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:19.011647 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:19.011651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:19.011653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:19.012048 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:19.012403 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:19.012721 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 02:29:19.012963 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 02:29:19.014031 6331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:19.157132 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 6196
I20260808 02:29:19.181275 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 6344
Aug 08 02:29:19 dist-test-slave-pgkr krb5kdc[5937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:29:19.201741 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:19.182175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.19.231.3:32967)
0808 02:29:19.182397 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:19.182403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:19.182460 (+ 57us) server_negotiation.cc:407] Connection header received
0808 02:29:19.182681 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:19.182688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:19.182800 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:19.182947 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:19.184577 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:19.185783 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:19.187497 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:19.188212 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:19.188367 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:19.193639 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:19.193661 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:19.193686 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:19.193738 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:19.198151 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:19.199104 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:19.199114 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:19.199118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:19.199392 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:19.199873 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:19.199879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:19.199882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:19.200382 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:19.200729 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:19.201029 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 02:29:19.201287 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 02:29:19.202775 6478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:19.337303 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 6344
I20260808 02:29:19.362412 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 5951
I20260808 02:29:19.363720 6019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:19.479080 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 5951
2026-08-08T02:29:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4763 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T02:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:19Z Disabled control of system clock
I20260808 02:29:19.585913 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45733
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:45733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:29:19.941188 6541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:19.941483 6541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:19.941632 6541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:19.948915 6541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:19.949028 6541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:19.949070 6541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:19.949123 6541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:19.949172 6541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:19.958847 6541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:45733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:19.960228 6541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:19.962069 6541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:19.974112 6546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:19.974328 6547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:19.974452 6549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:19.975739 6541 server_base.cc:1034] running on GCE node
I20260808 02:29:19.976558 6541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:19.977998 6541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:19.979246 6541 hybrid_clock.cc:648] HybridClock initialized: now 1786156159979174 us; error 50 us; skew 500 ppm
I20260808 02:29:19.979619 6541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:19.981997 6541 webserver.cc:460] Webserver started at http://127.19.231.62:45537/ using document root <none> and password file <none>
I20260808 02:29:19.982810 6541 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:19.982947 6541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:19.983289 6541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:19.986229 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "b06bb99495634b0d94781df0ceaeed4e"
format_stamp: "Formatted at 2026-08-08 02:29:19 on dist-test-slave-pgkr"
server_key: "36cc6ecad8c2a93327059fa6be3ff010"
server_key_iv: "5d81376a8e0e2a56a9905ccfb03771ca"
server_key_version: "encryptionkey@0"
I20260808 02:29:19.987089 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "b06bb99495634b0d94781df0ceaeed4e"
format_stamp: "Formatted at 2026-08-08 02:29:19 on dist-test-slave-pgkr"
server_key: "36cc6ecad8c2a93327059fa6be3ff010"
server_key_iv: "5d81376a8e0e2a56a9905ccfb03771ca"
server_key_version: "encryptionkey@0"
I20260808 02:29:19.992285 6541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:29:19.995767 6555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:19.997063 6541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:29:19.997265 6541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "b06bb99495634b0d94781df0ceaeed4e"
format_stamp: "Formatted at 2026-08-08 02:29:19 on dist-test-slave-pgkr"
server_key: "36cc6ecad8c2a93327059fa6be3ff010"
server_key_iv: "5d81376a8e0e2a56a9905ccfb03771ca"
server_key_version: "encryptionkey@0"
I20260808 02:29:19.997490 6541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:20.012512 6541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:20.013351 6541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:20.013662 6541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:20.033861 6541 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:45733
I20260808 02:29:20.033903 6612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:45733 every 8 connection(s)
I20260808 02:29:20.035398 6541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:20.039565 6613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:20.045749 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 6541
I20260808 02:29:20.045930 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:20.046432 20380 external_mini_cluster.cc:1428] Setting key 1ce644e0f2e883190d2fb58c9415da3a
I20260808 02:29:20.050844 6613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e: Bootstrap starting.
I20260808 02:29:20.054884 6613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:20.056262 6613 log.cc:824] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:20.059934 6613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e: No bootstrap required, opened a new log
I20260808 02:29:20.061691 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:20.048584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.19.231.62:45733)
0808 02:29:20.049637 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:20.049649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:20.049692 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:29:20.051701 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:20.051726 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:20.052231 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:20.052642 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:20.053339 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:20.054584 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:20.056706 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:20.057888 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:20.058032 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:20.059340 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:20.059347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:20.059365 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:20.059889 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:20.059977 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:20.060031 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:20.060176 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:20.060439 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 02:29:20.060751 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":400,"threads_started":1}
I20260808 02:29:20.068349 6613 raft_consensus.cc:348] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bb99495634b0d94781df0ceaeed4e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45733 } }
I20260808 02:29:20.068678 6613 raft_consensus.cc:374] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:20.068779 6613 raft_consensus.cc:729] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06bb99495634b0d94781df0ceaeed4e, State: Initialized, Role: FOLLOWER
I20260808 02:29:20.069484 6613 consensus_queue.cc:260] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bb99495634b0d94781df0ceaeed4e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45733 } }
I20260808 02:29:20.069800 6613 raft_consensus.cc:388] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:20.069955 6613 raft_consensus.cc:482] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:20.070137 6613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:20.072458 6613 raft_consensus.cc:504] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bb99495634b0d94781df0ceaeed4e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45733 } }
I20260808 02:29:20.072911 6613 leader_election.cc:302] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06bb99495634b0d94781df0ceaeed4e; no voters:
I20260808 02:29:20.073632 6613 leader_election.cc:288] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:20.073781 6618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:20.074605 6618 raft_consensus.cc:686] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [term 1 LEADER]: Becoming Leader. State: Replica: b06bb99495634b0d94781df0ceaeed4e, State: Running, Role: LEADER
I20260808 02:29:20.075225 6618 consensus_queue.cc:237] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bb99495634b0d94781df0ceaeed4e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45733 } }
I20260808 02:29:20.075911 6613 sys_catalog.cc:564] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:20.078080 6620 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b06bb99495634b0d94781df0ceaeed4e. Latest consensus state: current_term: 1 leader_uuid: "b06bb99495634b0d94781df0ceaeed4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bb99495634b0d94781df0ceaeed4e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45733 } } }
I20260808 02:29:20.078294 6619 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06bb99495634b0d94781df0ceaeed4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06bb99495634b0d94781df0ceaeed4e" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45733 } } }
I20260808 02:29:20.078549 6619 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:20.078547 6620 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:20.080890 6625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:20.086411 6625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:20.095881 6625 catalog_manager.cc:1347] Generated new cluster ID: 5316a30abf3d43b99bc7925957b26084
I20260808 02:29:20.096009 6625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:20.115962 6625 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:20.117349 6625 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:20.131211 6625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e: Generated new TSK 0
I20260808 02:29:20.132131 6625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:20.138991 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45733
--builtin_ntp_servers=127.19.231.20:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:29:20.499413 6637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:20.499696 6637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:20.499853 6637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:20.507375 6637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:20.507488 6637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:20.507681 6637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:29:20.516917 6637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:20.518373 6637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:20.520221 6637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:20.533726 6643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:20.533812 6642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:20.533829 6645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:20.540005 6637 server_base.cc:1034] running on GCE node
I20260808 02:29:20.540665 6637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:20.541674 6637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:20.542908 6637 hybrid_clock.cc:648] HybridClock initialized: now 1786156160542850 us; error 50 us; skew 500 ppm
I20260808 02:29:20.543260 6637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:20.545583 6637 webserver.cc:460] Webserver started at http://127.19.231.1:43691/ using document root <none> and password file <none>
I20260808 02:29:20.546319 6637 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:20.546460 6637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:20.546862 6637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:20.549856 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "515dbb94a657455bb3543111e7cac258"
format_stamp: "Formatted at 2026-08-08 02:29:20 on dist-test-slave-pgkr"
server_key: "df2fd58fbaba54b7129a245e97f302b4"
server_key_iv: "4b7b433965f1a185a17eb5b55a517101"
server_key_version: "encryptionkey@0"
I20260808 02:29:20.550736 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "515dbb94a657455bb3543111e7cac258"
format_stamp: "Formatted at 2026-08-08 02:29:20 on dist-test-slave-pgkr"
server_key: "df2fd58fbaba54b7129a245e97f302b4"
server_key_iv: "4b7b433965f1a185a17eb5b55a517101"
server_key_version: "encryptionkey@0"
I20260808 02:29:20.556255 6637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:29:20.559723 6651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:20.560999 6637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:29:20.561178 6637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "515dbb94a657455bb3543111e7cac258"
format_stamp: "Formatted at 2026-08-08 02:29:20 on dist-test-slave-pgkr"
server_key: "df2fd58fbaba54b7129a245e97f302b4"
server_key_iv: "4b7b433965f1a185a17eb5b55a517101"
server_key_version: "encryptionkey@0"
I20260808 02:29:20.561409 6637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:20.580410 6637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:20.581271 6637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:20.581599 6637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:20.582594 6637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:20.584100 6637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:20.584242 6637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:20.584376 6637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:20.584470 6637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:20.620456 6637 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:46201
I20260808 02:29:20.620507 6776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:46201 every 8 connection(s)
I20260808 02:29:20.621829 6637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:29:20.623965 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 6637
I20260808 02:29:20.624193 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:29:20.624831 20380 external_mini_cluster.cc:1428] Setting key f505ffa590907e9d38b00e74bdd9289e
I20260808 02:29:20.628422 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45733
--builtin_ntp_servers=127.19.231.20:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:29:20.638969 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:20.624996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:53697 (local address 127.19.231.62:45733)
0808 02:29:20.625557 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:20.625564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:20.628783 (+ 3219us) server_negotiation.cc:407] Connection header received
0808 02:29:20.629773 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:20.629778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:20.629858 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:20.629990 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:20.631552 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:20.632723 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:20.635330 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:20.636061 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:20.636258 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:20.637912 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:20.637918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:20.637921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:20.637946 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:20.638020 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:20.638036 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:20.638149 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:20.638419 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 02:29:20.638541 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260808 02:29:20.640226 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:20.625684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45733 (local address 127.19.231.1:53697)
0808 02:29:20.626868 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:20.626928 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:20.629456 (+ 2528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:20.630256 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:20.630272 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:20.630401 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:20.631320 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:20.631349 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:20.632904 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:20.632913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:20.635134 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:20.635153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:20.636535 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:20.636541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:20.636765 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:20.637552 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:20.637555 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:20.637598 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:20.637688 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:20.638152 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:20.638176 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:20.638185 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:29:20.638389 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 02:29:20.638881 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":371,"threads_started":1}
I20260808 02:29:20.642059 6777 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45733
I20260808 02:29:20.642441 6777 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:20.643502 6777 heartbeater.cc:511] Master 127.19.231.62:45733 requested a full tablet report, sending...
I20260808 02:29:20.645913 6575 ts_manager.cc:194] Registered new tserver with Master: 515dbb94a657455bb3543111e7cac258 (127.19.231.1:46201)
I20260808 02:29:20.648181 6575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.1:53697
W20260808 02:29:20.988119 6781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:20.988397 6781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:20.988487 6781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:20.995843 6781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:20.995956 6781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:20.996065 6781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:29:21.005473 6781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:21.006819 6781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:21.008627 6781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:21.022042 6786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:21.022121 6789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:21.022152 6787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:21.023496 6781 server_base.cc:1034] running on GCE node
I20260808 02:29:21.024108 6781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:21.025142 6781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:21.026360 6781 hybrid_clock.cc:648] HybridClock initialized: now 1786156161026289 us; error 61 us; skew 500 ppm
I20260808 02:29:21.026784 6781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:21.029101 6781 webserver.cc:460] Webserver started at http://127.19.231.2:36087/ using document root <none> and password file <none>
I20260808 02:29:21.029826 6781 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:21.029919 6781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:21.030294 6781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:21.033456 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "5688ec6b96304faca3e16ff71d5674ac"
format_stamp: "Formatted at 2026-08-08 02:29:21 on dist-test-slave-pgkr"
server_key: "ad3a1b3dc1711acc62c07e59c30f3d4b"
server_key_iv: "78b0e7b3a192597952951c4a5774d0bd"
server_key_version: "encryptionkey@0"
I20260808 02:29:21.034277 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "5688ec6b96304faca3e16ff71d5674ac"
format_stamp: "Formatted at 2026-08-08 02:29:21 on dist-test-slave-pgkr"
server_key: "ad3a1b3dc1711acc62c07e59c30f3d4b"
server_key_iv: "78b0e7b3a192597952951c4a5774d0bd"
server_key_version: "encryptionkey@0"
I20260808 02:29:21.039709 6781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:29:21.043476 6795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:21.044982 6781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:29:21.045150 6781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "5688ec6b96304faca3e16ff71d5674ac"
format_stamp: "Formatted at 2026-08-08 02:29:21 on dist-test-slave-pgkr"
server_key: "ad3a1b3dc1711acc62c07e59c30f3d4b"
server_key_iv: "78b0e7b3a192597952951c4a5774d0bd"
server_key_version: "encryptionkey@0"
I20260808 02:29:21.045341 6781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:21.058895 6781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:21.059738 6781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:21.060063 6781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:21.061092 6781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:21.062598 6781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:21.062793 6781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:21.062930 6781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:21.063015 6781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:21.099476 6781 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:37773
I20260808 02:29:21.099594 6921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:37773 every 8 connection(s)
I20260808 02:29:21.100876 6781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:29:21.106477 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 6781
I20260808 02:29:21.106706 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:29:21.107304 20380 external_mini_cluster.cc:1428] Setting key 87103117eb5b30e648ea5473e9251761
I20260808 02:29:21.110311 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45733
--builtin_ntp_servers=127.19.231.20:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:29:21.115875 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.103901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:37927 (local address 127.19.231.62:45733)
0808 02:29:21.104114 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.104119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.105534 (+ 1415us) server_negotiation.cc:407] Connection header received
0808 02:29:21.108046 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.108052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.108163 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.108271 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:21.109616 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.110340 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.112931 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.113354 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.113454 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.115150 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:21.115154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:21.115157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:21.115175 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:21.115207 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:21.115219 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:21.115290 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.115563 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.115652 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:29:21.117403 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.104431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45733 (local address 127.19.231.2:37927)
0808 02:29:21.105359 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:21.105402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:21.107708 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:21.108559 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:21.108568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:21.108667 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:21.109413 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.109442 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.110538 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.110546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.112773 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.112791 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.113589 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.113599 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.113937 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.114790 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:21.114796 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:21.114850 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:21.114961 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:21.115333 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:21.115346 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:21.115356 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 02:29:21.115532 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 02:29:21.115949 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
I20260808 02:29:21.119089 6923 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45733
I20260808 02:29:21.119453 6923 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:21.120242 6923 heartbeater.cc:511] Master 127.19.231.62:45733 requested a full tablet report, sending...
I20260808 02:29:21.121718 6575 ts_manager.cc:194] Registered new tserver with Master: 5688ec6b96304faca3e16ff71d5674ac (127.19.231.2:37773)
I20260808 02:29:21.122967 6575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.2:37927
W20260808 02:29:21.484261 6926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:21.484629 6926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:21.484815 6926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:21.492117 6926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:21.492225 6926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:21.492463 6926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:29:21.501924 6926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:45733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:21.503366 6926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:21.505378 6926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:21.519193 6932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:21.519224 6935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:21.519503 6931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:21.520455 6926 server_base.cc:1034] running on GCE node
I20260808 02:29:21.521067 6926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:21.522081 6926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:21.523311 6926 hybrid_clock.cc:648] HybridClock initialized: now 1786156161523255 us; error 42 us; skew 500 ppm
I20260808 02:29:21.523639 6926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:21.526111 6926 webserver.cc:460] Webserver started at http://127.19.231.3:40059/ using document root <none> and password file <none>
I20260808 02:29:21.526885 6926 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:21.526991 6926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:21.527285 6926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:21.530256 6926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "bfd353d3a9064d73b850fa520ecad04b"
format_stamp: "Formatted at 2026-08-08 02:29:21 on dist-test-slave-pgkr"
server_key: "12ac9af031ccb2d25e4369b958bf5d86"
server_key_iv: "2748fbc3dddc347309366ee6e7e12a3b"
server_key_version: "encryptionkey@0"
I20260808 02:29:21.531131 6926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "bfd353d3a9064d73b850fa520ecad04b"
format_stamp: "Formatted at 2026-08-08 02:29:21 on dist-test-slave-pgkr"
server_key: "12ac9af031ccb2d25e4369b958bf5d86"
server_key_iv: "2748fbc3dddc347309366ee6e7e12a3b"
server_key_version: "encryptionkey@0"
I20260808 02:29:21.536726 6926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:29:21.540130 6941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:21.541429 6926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:29:21.541565 6926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "bfd353d3a9064d73b850fa520ecad04b"
format_stamp: "Formatted at 2026-08-08 02:29:21 on dist-test-slave-pgkr"
server_key: "12ac9af031ccb2d25e4369b958bf5d86"
server_key_iv: "2748fbc3dddc347309366ee6e7e12a3b"
server_key_version: "encryptionkey@0"
I20260808 02:29:21.541745 6926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:21.568521 6926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:21.569402 6926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:21.569717 6926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:21.570784 6926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:21.572460 6926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:21.572566 6926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:21.572662 6926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:21.572722 6926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:21.608728 6926 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:40203
I20260808 02:29:21.608810 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:40203 every 8 connection(s)
I20260808 02:29:21.610189 6926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:29:21.616671 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 6926
I20260808 02:29:21.616876 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:29:21.617538 20380 external_mini_cluster.cc:1428] Setting key 3886b0da1be698f874694393729577ac
I20260808 02:29:21.624189 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.613017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:58567 (local address 127.19.231.62:45733)
0808 02:29:21.613206 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.613211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.614736 (+ 1525us) server_negotiation.cc:407] Connection header received
0808 02:29:21.616814 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.616820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.616916 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.617074 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:21.618430 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.619360 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.621267 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.621874 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.621992 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.623353 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:21.623360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:21.623364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:21.623385 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:21.623425 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:21.623440 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:21.623536 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.623737 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.623830 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:29:21.625135 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.613569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:45733 (local address 127.19.231.3:58567)
0808 02:29:21.614562 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:21.614603 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:21.616503 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:21.617320 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:21.617331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:21.617455 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:21.618218 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.618238 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.619536 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.619541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.621120 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.621134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.622089 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.622094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.622299 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.623042 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:21.623045 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:21.623084 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:21.623161 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:21.623539 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:21.623548 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:21.623554 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 02:29:21.623693 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 02:29:21.624060 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":294,"threads_started":1}
I20260808 02:29:21.626493 7066 heartbeater.cc:347] Connected to a master server at 127.19.231.62:45733
I20260808 02:29:21.626825 7066 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:21.627506 7066 heartbeater.cc:511] Master 127.19.231.62:45733 requested a full tablet report, sending...
I20260808 02:29:21.628804 6575 ts_manager.cc:194] Registered new tserver with Master: bfd353d3a9064d73b850fa520ecad04b (127.19.231.3:40203)
I20260808 02:29:21.629614 6575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.3:58567
I20260808 02:29:21.632946 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:29:21.643414 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.637191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.19.231.62:45733)
0808 02:29:21.637353 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.637378 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.637811 (+ 433us) server_negotiation.cc:407] Connection header received
0808 02:29:21.638013 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.638018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.638088 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.638192 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:21.638794 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.639481 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.641013 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.641432 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.641571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.642658 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:21.642665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:21.642670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:21.642701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:21.642734 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:21.642747 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:21.642846 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.642998 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.643095 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:29:21.651427 6777 heartbeater.cc:503] Master 127.19.231.62:45733 was elected leader, sending a full tablet report...
I20260808 02:29:21.647472 6575 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:29:21.655267 6575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:29:21.681016 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.668397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40074)
0808 02:29:21.670304 (+ 1907us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:21.670325 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:21.670446 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:21.672579 (+ 2133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:21.672591 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:21.672635 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:21.673130 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.673146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.675317 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.675323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.677844 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.677856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.679832 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.679842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.680072 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.680135 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 02:29:21.680266 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 02:29:21.680488 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1751,"mutex_wait_us":1,"thread_start_us":4992,"threads_started":1}
I20260808 02:29:21.681516 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.668174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:40203 (local address 127.0.0.1:55056)
0808 02:29:21.669126 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:21.669172 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:21.669373 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:21.674029 (+ 4656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:21.674033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:21.674069 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:21.674384 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.674394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.676165 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.676172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.678593 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.678630 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.680624 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.680631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.680823 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.680860 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:29:21.680938 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 02:29:21.681117 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":23,"thread_start_us":470,"threads_started":1}
I20260808 02:29:21.681881 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.669713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:37773 (local address 127.0.0.1:35872)
0808 02:29:21.673673 (+ 3960us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:21.673691 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:21.673814 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:21.674528 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:21.674535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:21.674564 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:21.675000 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.675009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.677207 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.677214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.679225 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.679236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.681063 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.681070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.681241 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.681276 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 02:29:21.681347 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:29:21.681529 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3842,"mutex_wait_us":23,"thread_start_us":917,"threads_started":1}
I20260808 02:29:21.681936 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.670369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.19.231.2:37773)
0808 02:29:21.671053 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.671065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.673961 (+ 2896us) server_negotiation.cc:407] Connection header received
0808 02:29:21.674085 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.674095 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.674337 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.674556 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:21.675187 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.677055 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.679407 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.680844 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.680999 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.681330 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.681422 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.681591 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":421,"threads_started":1}
I20260808 02:29:21.682964 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.669089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.19.231.1:46201)
0808 02:29:21.671659 (+ 2570us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.671673 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.671714 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:29:21.671860 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.671873 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.672241 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.672539 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:21.673417 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.675100 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.678057 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.679560 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.681895 (+ 2335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.682218 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.682312 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.682506 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2326,"thread_start_us":2150,"threads_started":1}
I20260808 02:29:21.683022 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.668426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.19.231.3:40203)
0808 02:29:21.673403 (+ 4977us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.673413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.673456 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:29:21.673563 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.673571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.673830 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.674027 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:21.674590 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.675977 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.678826 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.680388 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.682122 (+ 1734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.682389 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.682453 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.682637 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4774,"thread_start_us":4595,"threads_started":1}
I20260808 02:29:21.686234 6703 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8c6087018d474ba344d23c41a9fbbc (DEFAULT_TABLE table=test-table [id=635bd11dc6474b95a1b6679a9867afa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:21.686242 6849 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8c6087018d474ba344d23c41a9fbbc (DEFAULT_TABLE table=test-table [id=635bd11dc6474b95a1b6679a9867afa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:21.687595 6703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8c6087018d474ba344d23c41a9fbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:21.687755 6849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8c6087018d474ba344d23c41a9fbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:21.689612 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8c6087018d474ba344d23c41a9fbbc (DEFAULT_TABLE table=test-table [id=635bd11dc6474b95a1b6679a9867afa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:21.691197 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8c6087018d474ba344d23c41a9fbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:21.699206 7086 tablet_bootstrap.cc:492] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: Bootstrap starting.
I20260808 02:29:21.703528 7086 tablet_bootstrap.cc:654] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:21.703961 7087 tablet_bootstrap.cc:492] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Bootstrap starting.
I20260808 02:29:21.704967 7086 log.cc:824] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:21.707319 7089 tablet_bootstrap.cc:492] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Bootstrap starting.
I20260808 02:29:21.707650 7087 tablet_bootstrap.cc:654] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:21.708407 7086 tablet_bootstrap.cc:492] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: No bootstrap required, opened a new log
I20260808 02:29:21.708701 7086 ts_tablet_manager.cc:1397] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260808 02:29:21.708864 7087 log.cc:824] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:21.710896 7089 tablet_bootstrap.cc:654] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:21.712146 7089 log.cc:824] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:21.712247 7087 tablet_bootstrap.cc:492] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: No bootstrap required, opened a new log
I20260808 02:29:21.712570 7087 ts_tablet_manager.cc:1397] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 02:29:21.719655 7086 raft_consensus.cc:348] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.720115 7086 raft_consensus.cc:374] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:21.720291 7086 raft_consensus.cc:729] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515dbb94a657455bb3543111e7cac258, State: Initialized, Role: FOLLOWER
I20260808 02:29:21.721004 7086 consensus_queue.cc:260] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.723093 7087 raft_consensus.cc:348] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.723466 7087 raft_consensus.cc:374] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:21.723618 7087 raft_consensus.cc:729] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5688ec6b96304faca3e16ff71d5674ac, State: Initialized, Role: FOLLOWER
I20260808 02:29:21.723873 7086 ts_tablet_manager.cc:1428] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260808 02:29:21.724020 7089 tablet_bootstrap.cc:492] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: No bootstrap required, opened a new log
I20260808 02:29:21.724340 7089 ts_tablet_manager.cc:1397] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260808 02:29:21.724383 7087 consensus_queue.cc:260] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.725543 6923 heartbeater.cc:503] Master 127.19.231.62:45733 was elected leader, sending a full tablet report...
I20260808 02:29:21.726188 7087 ts_tablet_manager.cc:1428] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 02:29:21.733289 7089 raft_consensus.cc:348] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.733652 7089 raft_consensus.cc:374] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:21.733773 7089 raft_consensus.cc:729] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd353d3a9064d73b850fa520ecad04b, State: Initialized, Role: FOLLOWER
I20260808 02:29:21.734347 7089 consensus_queue.cc:260] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.735466 7066 heartbeater.cc:503] Master 127.19.231.62:45733 was elected leader, sending a full tablet report...
I20260808 02:29:21.736085 7089 ts_tablet_manager.cc:1428] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260808 02:29:21.840395 7095 raft_consensus.cc:482] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:21.840699 7095 raft_consensus.cc:504] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.842489 7095 leader_election.cc:288] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 515dbb94a657455bb3543111e7cac258 (127.19.231.1:46201), 5688ec6b96304faca3e16ff71d5674ac (127.19.231.2:37773)
I20260808 02:29:21.850260 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.843029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:40347 (local address 127.19.231.1:46201)
0808 02:29:21.843363 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.843369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.843390 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:29:21.843681 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.843689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.843793 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.843912 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:21.844535 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.845767 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.848275 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.849439 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.849600 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.849676 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.849838 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.849974 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260808 02:29:21.850282 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.842937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.19.231.3:40347)
0808 02:29:21.843264 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:21.843287 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:21.843422 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:21.844050 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:21.844058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:21.844096 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:21.844334 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.844349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.845905 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.845911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.848079 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.848094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.849571 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.849575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.849688 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.849714 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 02:29:21.849790 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 02:29:21.849885 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":32}
I20260808 02:29:21.850950 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.843206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:34571 (local address 127.19.231.2:37773)
0808 02:29:21.843439 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.843448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.844123 (+ 675us) server_negotiation.cc:407] Connection header received
0808 02:29:21.844525 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.844587 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.844754 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.844880 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:21.845563 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.846363 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.848710 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.850000 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.850153 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.850210 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.850596 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.850743 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 02:29:21.851078 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.843149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:37773 (local address 127.19.231.3:34571)
0808 02:29:21.844005 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:21.844025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:21.844252 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:21.844924 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:21.844929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:21.844955 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:21.845384 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.845397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.846526 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.846534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.848548 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:21.848559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.850177 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.850182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:21.850381 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.850419 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 02:29:21.850561 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 02:29:21.850731 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":33,"thread_start_us":495,"threads_started":1}
I20260808 02:29:21.851425 6727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c6087018d474ba344d23c41a9fbbc" candidate_uuid: "bfd353d3a9064d73b850fa520ecad04b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515dbb94a657455bb3543111e7cac258" is_pre_election: true
I20260808 02:29:21.851948 6727 raft_consensus.cc:2454] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd353d3a9064d73b850fa520ecad04b in term 0.
I20260808 02:29:21.851971 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c6087018d474ba344d23c41a9fbbc" candidate_uuid: "bfd353d3a9064d73b850fa520ecad04b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688ec6b96304faca3e16ff71d5674ac" is_pre_election: true
I20260808 02:29:21.852494 6871 raft_consensus.cc:2454] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd353d3a9064d73b850fa520ecad04b in term 0.
I20260808 02:29:21.852732 6943 leader_election.cc:302] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515dbb94a657455bb3543111e7cac258, bfd353d3a9064d73b850fa520ecad04b; no voters:
I20260808 02:29:21.853153 7095 raft_consensus.cc:2781] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 02:29:21.853269 6924 tablet.cc:2369] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:21.853327 7095 raft_consensus.cc:482] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:29:21.853426 7095 raft_consensus.cc:3037] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:21.855947 7095 raft_consensus.cc:504] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
I20260808 02:29:21.856571 7095 leader_election.cc:288] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [CANDIDATE]: Term 1 election: Requested vote from peers 515dbb94a657455bb3543111e7cac258 (127.19.231.1:46201), 5688ec6b96304faca3e16ff71d5674ac (127.19.231.2:37773)
I20260808 02:29:21.856961 6727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c6087018d474ba344d23c41a9fbbc" candidate_uuid: "bfd353d3a9064d73b850fa520ecad04b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515dbb94a657455bb3543111e7cac258"
I20260808 02:29:21.857031 6871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8c6087018d474ba344d23c41a9fbbc" candidate_uuid: "bfd353d3a9064d73b850fa520ecad04b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5688ec6b96304faca3e16ff71d5674ac"
I20260808 02:29:21.857183 6727 raft_consensus.cc:3037] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:21.857240 6871 raft_consensus.cc:3037] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:21.859521 6727 raft_consensus.cc:2454] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd353d3a9064d73b850fa520ecad04b in term 1.
I20260808 02:29:21.859619 6871 raft_consensus.cc:2454] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd353d3a9064d73b850fa520ecad04b in term 1.
I20260808 02:29:21.859931 6943 leader_election.cc:302] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515dbb94a657455bb3543111e7cac258, bfd353d3a9064d73b850fa520ecad04b; no voters:
I20260808 02:29:21.860296 7095 raft_consensus.cc:2781] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:21.860687 7095 raft_consensus.cc:686] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 1 LEADER]: Becoming Leader. State: Replica: bfd353d3a9064d73b850fa520ecad04b, State: Running, Role: LEADER
I20260808 02:29:21.861270 7095 consensus_queue.cc:237] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } }
W20260808 02:29:21.862242 7067 tablet.cc:2369] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:21.866569 6574 catalog_manager.cc:5495] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b reported cstate change: term changed from 0 to 1, leader changed from <none> to bfd353d3a9064d73b850fa520ecad04b (127.19.231.3). New cstate: current_term: 1 leader_uuid: "bfd353d3a9064d73b850fa520ecad04b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd353d3a9064d73b850fa520ecad04b" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 40203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 } health_report { overall_health: UNKNOWN } } }
W20260808 02:29:21.874329 6778 tablet.cc:2369] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:21.921527 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.916087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.19.231.3:40203)
0808 02:29:21.916303 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.916312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.916425 (+ 113us) server_negotiation.cc:407] Connection header received
0808 02:29:21.916607 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.916611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.916698 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.916844 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:29:21.917204 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.917986 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.919475 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.919907 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.920019 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.920471 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:29:21.920981 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:29:21.921116 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.921156 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.921221 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:29:21.929914 6871 raft_consensus.cc:1261] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 1 FOLLOWER]: Refusing update from remote peer bfd353d3a9064d73b850fa520ecad04b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:21.929919 6727 raft_consensus.cc:1261] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 1 FOLLOWER]: Refusing update from remote peer bfd353d3a9064d73b850fa520ecad04b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:21.930950 7097 consensus_queue.cc:1035] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5688ec6b96304faca3e16ff71d5674ac" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 37773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:21.931341 7095 consensus_queue.cc:1035] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [LEADER]: Connected to new peer: Peer: permanent_uuid: "515dbb94a657455bb3543111e7cac258" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 46201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:21.944181 7101 mvcc.cc:205] Tried to move back new op lower bound from 7316095639254777856 to 7316095638988705792. Current Snapshot: MvccSnapshot[applied={T|T < 7316095639254777856}]
I20260808 02:29:21.957155 6574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:21.957479 6574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:21.961354 6574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b06bb99495634b0d94781df0ceaeed4e: Sending DeleteTablet for 3 replicas of tablet 5d8c6087018d474ba344d23c41a9fbbc
I20260808 02:29:21.962464 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8c6087018d474ba344d23c41a9fbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:21 UTC) from {username='slave'} at 127.0.0.1:55056
I20260808 02:29:21.962579 6849 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8c6087018d474ba344d23c41a9fbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:21 UTC) from {username='slave'} at 127.0.0.1:35872
I20260808 02:29:21.962594 6703 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8c6087018d474ba344d23c41a9fbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:21 UTC) from {username='slave'} at 127.0.0.1:40074
I20260808 02:29:21.966673 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 6637
I20260808 02:29:21.980480 7112 tablet_replica.cc:330] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: stopping tablet replica
I20260808 02:29:21.981144 7112 raft_consensus.cc:2229] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:21.982059 7113 tablet_replica.cc:330] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258: stopping tablet replica
I20260808 02:29:21.982649 7113 raft_consensus.cc:2229] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:21.983470 7110 tablet_replica.cc:330] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: stopping tablet replica
I20260808 02:29:21.983887 7110 raft_consensus.cc:2229] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:29:21.984663 7113 raft_consensus.cc:2258] T 5d8c6087018d474ba344d23c41a9fbbc P 515dbb94a657455bb3543111e7cac258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:21.984725 7112 raft_consensus.cc:2258] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:21.985323 7110 raft_consensus.cc:2258] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:21.987330 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:21.967988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.19.231.1:46201)
0808 02:29:21.968166 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:21.968173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:21.971461 (+ 3288us) server_negotiation.cc:407] Connection header received
0808 02:29:21.972510 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:21.972520 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:21.972690 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:21.972795 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:21.974312 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.975936 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.982966 (+ 7030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:21.983510 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:21.983678 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:21.985490 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:21.985502 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:21.985519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:21.986322 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:21.986386 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:21.986412 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:21.986541 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:21.986848 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 02:29:21.986974 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 02:29:21.988176 7110 ts_tablet_manager.cc:1905] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:21.988476 6772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:21.992762 7112 ts_tablet_manager.cc:1905] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:21.997033 7110 ts_tablet_manager.cc:1918] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:21.997231 7110 log.cc:1196] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/5d8c6087018d474ba344d23c41a9fbbc
I20260808 02:29:21.997723 7112 ts_tablet_manager.cc:1918] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:21.997836 7110 ts_tablet_manager.cc:1939] T 5d8c6087018d474ba344d23c41a9fbbc P bfd353d3a9064d73b850fa520ecad04b: Deleting consensus metadata
I20260808 02:29:21.997893 7112 log.cc:1196] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/5d8c6087018d474ba344d23c41a9fbbc
I20260808 02:29:21.998435 7112 ts_tablet_manager.cc:1939] T 5d8c6087018d474ba344d23c41a9fbbc P 5688ec6b96304faca3e16ff71d5674ac: Deleting consensus metadata
I20260808 02:29:21.999814 6556 catalog_manager.cc:4841] TS bfd353d3a9064d73b850fa520ecad04b (127.19.231.3:40203): tablet 5d8c6087018d474ba344d23c41a9fbbc (table test-table [id=635bd11dc6474b95a1b6679a9867afa9]) successfully deleted
I20260808 02:29:22.000764 6556 catalog_manager.cc:4841] TS 5688ec6b96304faca3e16ff71d5674ac (127.19.231.2:37773): tablet 5d8c6087018d474ba344d23c41a9fbbc (table test-table [id=635bd11dc6474b95a1b6679a9867afa9]) successfully deleted
I20260808 02:29:22.087141 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 6637
I20260808 02:29:22.111737 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 6781
W20260808 02:29:22.111784 6557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.231.1:46201 (error 108)
I20260808 02:29:22.125217 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.113515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.19.231.2:37773)
0808 02:29:22.113763 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:22.113771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:22.113795 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:29:22.113874 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:22.113882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:22.114008 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:22.114146 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:22.115569 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:22.117180 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:22.119822 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:22.120772 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:22.120927 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:22.122770 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:22.122785 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:22.122810 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:22.123861 (+ 1051us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:22.123924 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:22.123952 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:22.124080 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:22.124745 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 02:29:22.124856 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:29:22.125370 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.124843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40086)
0808 02:29:22.125035 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.125191 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:29:22.125759 6557 catalog_manager.cc:4584] TS 515dbb94a657455bb3543111e7cac258 (127.19.231.1:46201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d8c6087018d474ba344d23c41a9fbbc: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
I20260808 02:29:22.126413 6916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:22.179502 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.178952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40096)
0808 02:29:22.179183 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.179305 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 02:29:22.180665 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.180245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40110)
0808 02:29:22.180412 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.180503 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 02:29:22.220844 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 6781
I20260808 02:29:22.241564 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.240934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40126)
0808 02:29:22.241219 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.241351 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 02:29:22.242992 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.242489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40140)
0808 02:29:22.242711 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.242826 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 02:29:22.245543 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 6926
I20260808 02:29:22.254279 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.246366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.19.231.3:40203)
0808 02:29:22.246554 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:22.246561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:22.246583 (+ 22us) server_negotiation.cc:407] Connection header received
0808 02:29:22.246829 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:22.246836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:22.246946 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:22.247056 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:22.247562 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:22.248765 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:22.250352 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:22.251166 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:22.251321 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:22.252479 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:22.252493 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:22.252513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:22.253407 (+ 894us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:22.253472 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:22.253497 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:22.253614 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:22.253745 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 02:29:22.253852 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:29:22.255251 7060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:22.323818 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.323256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40142)
0808 02:29:22.323515 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.323620 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260808 02:29:22.324961 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.324556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40152)
0808 02:29:22.324787 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.324853 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 02:29:22.381511 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 6926
I20260808 02:29:22.406029 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 6541
I20260808 02:29:22.407356 6608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:22.520766 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:22.520043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:46201 (local address 127.0.0.1:40168)
0808 02:29:22.520334 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:22.520476 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260808 02:29:22.520979 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 6541
2026-08-08T02:29:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3034 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:22 dist-test-slave-pgkr krb5kdc[7123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:22 dist-test-slave-pgkr krb5kdc[7123](info): set up 2 sockets
Aug 08 02:29:22 dist-test-slave-pgkr krb5kdc[7123](info): commencing operation
krb5kdc: starting...
W20260808 02:29:23.706842 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:23 dist-test-slave-pgkr krb5kdc[7123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:29:24.060491 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:33557
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:24.421911 7137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:24.422267 7137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:24.422435 7137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:24.430115 7137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:24.430277 7137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:24.430392 7137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:24.430501 7137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:24.430606 7137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:24.441120 7137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:24.442581 7137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:24.443981 7137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:24.456130 7145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:24.456108 7148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:24.455965 7146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:24.457321 7137 server_base.cc:1034] running on GCE node
I20260808 02:29:24.458168 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:24.459800 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:24.461133 7137 hybrid_clock.cc:648] HybridClock initialized: now 1786156164461026 us; error 47 us; skew 500 ppm
I20260808 02:29:24.461529 7137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:24 dist-test-slave-pgkr krb5kdc[7123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156164, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:24.464936 7137 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:29:24.466517 7137 webserver.cc:460] Webserver started at http://127.19.231.62:39581/ using document root <none> and password file <none>
I20260808 02:29:24.467361 7137 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:24.467525 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:24.467847 7137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:24.471060 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "4a693df407e84129a31bd2c33b3a9ba6"
format_stamp: "Formatted at 2026-08-08 02:29:24 on dist-test-slave-pgkr"
server_key: "5ca00507e4291734fd91c03fe20ced86"
server_key_iv: "2d466a94bd9822392300792b6563065c"
server_key_version: "encryptionkey@0"
I20260808 02:29:24.471823 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "4a693df407e84129a31bd2c33b3a9ba6"
format_stamp: "Formatted at 2026-08-08 02:29:24 on dist-test-slave-pgkr"
server_key: "5ca00507e4291734fd91c03fe20ced86"
server_key_iv: "2d466a94bd9822392300792b6563065c"
server_key_version: "encryptionkey@0"
I20260808 02:29:24.477263 7137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:29:24.480862 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:24.482334 7137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:29:24.482553 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "4a693df407e84129a31bd2c33b3a9ba6"
format_stamp: "Formatted at 2026-08-08 02:29:24 on dist-test-slave-pgkr"
server_key: "5ca00507e4291734fd91c03fe20ced86"
server_key_iv: "2d466a94bd9822392300792b6563065c"
server_key_version: "encryptionkey@0"
I20260808 02:29:24.482810 7137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:24.517009 7137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:24.527990 7137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:24.528330 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:24.548523 7137 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:33557
I20260808 02:29:24.548528 7213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:33557 every 8 connection(s)
I20260808 02:29:24.549932 7137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:24.553778 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 7137
I20260808 02:29:24.553992 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:24.554493 20380 external_mini_cluster.cc:1428] Setting key 768a2f2dce033d1ed7bbea15c826c7ac
I20260808 02:29:24.555008 7214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 02:29:24 dist-test-slave-pgkr krb5kdc[7123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:24.573297 7214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6: Bootstrap starting.
I20260808 02:29:24.577327 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:24.556412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.19.231.62:33557)
0808 02:29:24.557226 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:24.557238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:24.557278 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:29:24.558940 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:24.558963 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:24.559341 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:24.559760 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:24.561579 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:24.563109 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:24.564531 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:24.565199 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:24.565397 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:24.569823 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:24.569831 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:24.569844 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:24.569886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:24.573090 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:24.573864 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:24.573879 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:24.573889 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:24.574057 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:24.574450 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:24.574456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:24.574459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:24.575044 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:24.575401 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:24.575893 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 02:29:24.576339 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":388,"threads_started":1}
I20260808 02:29:24.577854 7214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:24.579348 7214 log.cc:824] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:24.582808 7214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6: No bootstrap required, opened a new log
I20260808 02:29:24.595813 7214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a693df407e84129a31bd2c33b3a9ba6" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33557 } }
I20260808 02:29:24.596350 7214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:24.596474 7214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a693df407e84129a31bd2c33b3a9ba6, State: Initialized, Role: FOLLOWER
I20260808 02:29:24.597214 7214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a693df407e84129a31bd2c33b3a9ba6" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33557 } }
I20260808 02:29:24.597518 7214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:24.597671 7214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:24.597867 7214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:24.601614 7214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a693df407e84129a31bd2c33b3a9ba6" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33557 } }
I20260808 02:29:24.602311 7214 leader_election.cc:302] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a693df407e84129a31bd2c33b3a9ba6; no voters:
I20260808 02:29:24.603072 7214 leader_election.cc:288] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:24.603144 7219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:24.603899 7219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 4a693df407e84129a31bd2c33b3a9ba6, State: Running, Role: LEADER
I20260808 02:29:24.604476 7219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a693df407e84129a31bd2c33b3a9ba6" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33557 } }
I20260808 02:29:24.605615 7214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:24.607211 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a693df407e84129a31bd2c33b3a9ba6. Latest consensus state: current_term: 1 leader_uuid: "4a693df407e84129a31bd2c33b3a9ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a693df407e84129a31bd2c33b3a9ba6" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33557 } } }
I20260808 02:29:24.607775 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:24.608640 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a693df407e84129a31bd2c33b3a9ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a693df407e84129a31bd2c33b3a9ba6" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 33557 } } }
I20260808 02:29:24.609048 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:24.610188 7225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:24.616796 7225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:24.625519 7225 catalog_manager.cc:1347] Generated new cluster ID: 7803cc71c0454c4a9d97f902589371b0
I20260808 02:29:24.625633 7225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:24.652027 7225 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:24.653885 7225 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:24.668430 7225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6: Generated new TSK 0
I20260808 02:29:24.669526 7225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:29:24.762234 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33557
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:25.129176 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:25.129498 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:25.129679 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:25.137076 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:25.137228 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:25.137416 7242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:29:25.147367 7242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:25.148729 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:25.150081 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:25.163790 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:25.163889 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:25.163862 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:25.164790 7242 server_base.cc:1034] running on GCE node
I20260808 02:29:25.165506 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:25.166455 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:25.167688 7242 hybrid_clock.cc:648] HybridClock initialized: now 1786156165167632 us; error 68 us; skew 500 ppm
I20260808 02:29:25.168081 7242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:25 dist-test-slave-pgkr krb5kdc[7123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156165, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:25.171432 7242 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:29:25.172956 7242 webserver.cc:460] Webserver started at http://127.19.231.1:34309/ using document root <none> and password file <none>
I20260808 02:29:25.173669 7242 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:25.173818 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:25.174154 7242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:25.177148 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "a24c8a682c734674b03db1a51dc5d4b9"
format_stamp: "Formatted at 2026-08-08 02:29:25 on dist-test-slave-pgkr"
server_key: "63654a34bdcfb44a8eb92a9630a515db"
server_key_iv: "2fa556992a6a9f60c758941368065e2d"
server_key_version: "encryptionkey@0"
I20260808 02:29:25.177915 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "a24c8a682c734674b03db1a51dc5d4b9"
format_stamp: "Formatted at 2026-08-08 02:29:25 on dist-test-slave-pgkr"
server_key: "63654a34bdcfb44a8eb92a9630a515db"
server_key_iv: "2fa556992a6a9f60c758941368065e2d"
server_key_version: "encryptionkey@0"
I20260808 02:29:25.183147 7242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:29:25.186811 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:25.188184 7242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:29:25.188352 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "a24c8a682c734674b03db1a51dc5d4b9"
format_stamp: "Formatted at 2026-08-08 02:29:25 on dist-test-slave-pgkr"
server_key: "63654a34bdcfb44a8eb92a9630a515db"
server_key_iv: "2fa556992a6a9f60c758941368065e2d"
server_key_version: "encryptionkey@0"
I20260808 02:29:25.188560 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:25.200073 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:25.203999 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:25.204319 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:25.205284 7242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:25.206804 7242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:25.206877 7242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:25.207034 7242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:25.207094 7242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:25.241472 7242 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:42871
I20260808 02:29:25.241523 7383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:42871 every 8 connection(s)
I20260808 02:29:25.242832 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:29:25.247401 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 7242
I20260808 02:29:25.247741 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:29:25.248550 20380 external_mini_cluster.cc:1428] Setting key 494f601e97e59e60a49300bc1a8f3ff1
Aug 08 02:29:25 dist-test-slave-pgkr krb5kdc[7123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156165, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:25.268021 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:25.245583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:57851 (local address 127.19.231.62:33557)
0808 02:29:25.246234 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:25.246243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:25.247501 (+ 1258us) server_negotiation.cc:407] Connection header received
0808 02:29:25.250351 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:25.250357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:25.250438 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:25.250543 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:25.252988 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.254184 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.256316 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.257020 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.257169 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:25.261659 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:25.261667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:25.261674 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:25.261717 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:25.264579 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:25.265554 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:25.265559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:25.265562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:25.265692 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:25.266213 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:25.266217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:25.266219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:25.266491 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:25.266687 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:25.267405 (+ 718us) server_negotiation.cc:299] Negotiation successful
0808 02:29:25.267639 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":304,"threads_started":1}
I20260808 02:29:25.269143 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:25.246294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:33557 (local address 127.19.231.1:57851)
0808 02:29:25.247306 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:25.247371 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:25.250001 (+ 2630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:25.251164 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:25.251176 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:25.251962 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:25.252803 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:25.252825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.254366 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.254370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:25.256149 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:25.256172 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.257248 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.257256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:25.257546 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:25.258304 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:25.258336 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:25.261395 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:25.264732 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:25.264745 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:25.264764 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:25.265422 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:25.265809 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:25.265815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:25.265820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:25.266064 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:25.266670 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:25.266678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:25.267020 (+ 342us) client_negotiation.cc:769] Sending connection context
0808 02:29:25.267289 (+ 269us) client_negotiation.cc:240] Negotiation successful
0808 02:29:25.267842 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":372,"threads_started":1}
I20260808 02:29:25.270838 7384 heartbeater.cc:347] Connected to a master server at 127.19.231.62:33557
I20260808 02:29:25.271209 7384 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:25.271951 7384 heartbeater.cc:511] Master 127.19.231.62:33557 requested a full tablet report, sending...
I20260808 02:29:25.274085 7174 ts_manager.cc:194] Registered new tserver with Master: a24c8a682c734674b03db1a51dc5d4b9 (127.19.231.1:42871)
I20260808 02:29:25.275619 7174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:57851
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:29:25.322175 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33557
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:25.675812 7390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:25.676095 7390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:25.676276 7390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:25.683674 7390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:25.683795 7390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:25.683979 7390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:29:25.694108 7390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:25.695441 7390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:25.696798 7390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:25.710191 7397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:25.710162 7401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:25.710297 7398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:25.712005 7390 server_base.cc:1034] running on GCE node
I20260808 02:29:25.712670 7390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:25.713670 7390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:25.714921 7390 hybrid_clock.cc:648] HybridClock initialized: now 1786156165714857 us; error 50 us; skew 500 ppm
I20260808 02:29:25.715250 7390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:25 dist-test-slave-pgkr krb5kdc[7123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156165, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:25.718394 7390 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:29:25.719964 7390 webserver.cc:460] Webserver started at http://127.19.231.2:33225/ using document root <none> and password file <none>
I20260808 02:29:25.720685 7390 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:25.720834 7390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:25.721154 7390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:25.724222 7390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "88b4eb73a33f4bfcb832227b8fd19183"
format_stamp: "Formatted at 2026-08-08 02:29:25 on dist-test-slave-pgkr"
server_key: "7452a635bdcccc202fa1bc4389c41b51"
server_key_iv: "4efc8bdedfbe2aaf2271ad06640edda1"
server_key_version: "encryptionkey@0"
I20260808 02:29:25.725036 7390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "88b4eb73a33f4bfcb832227b8fd19183"
format_stamp: "Formatted at 2026-08-08 02:29:25 on dist-test-slave-pgkr"
server_key: "7452a635bdcccc202fa1bc4389c41b51"
server_key_iv: "4efc8bdedfbe2aaf2271ad06640edda1"
server_key_version: "encryptionkey@0"
I20260808 02:29:25.730314 7390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:29:25.733942 7408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:25.735384 7390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:25.735603 7390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "88b4eb73a33f4bfcb832227b8fd19183"
format_stamp: "Formatted at 2026-08-08 02:29:25 on dist-test-slave-pgkr"
server_key: "7452a635bdcccc202fa1bc4389c41b51"
server_key_iv: "4efc8bdedfbe2aaf2271ad06640edda1"
server_key_version: "encryptionkey@0"
I20260808 02:29:25.735800 7390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:25.759096 7390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:25.763307 7390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:25.763659 7390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:25.764791 7390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:25.766239 7390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:25.766322 7390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:25.766492 7390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:25.766580 7390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:25.802029 7390 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:41197
I20260808 02:29:25.802091 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:41197 every 8 connection(s)
I20260808 02:29:25.803452 7390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:29:25.805509 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 7390
I20260808 02:29:25.805743 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:29:25.806295 20380 external_mini_cluster.cc:1428] Setting key 5e788c1f97e6e60a058b9669a3ee317b
Aug 08 02:29:25 dist-test-slave-pgkr krb5kdc[7123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156165, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:25.827530 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:25.806412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:59927 (local address 127.19.231.62:33557)
0808 02:29:25.810943 (+ 4531us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:25.810948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:25.810963 (+ 15us) server_negotiation.cc:407] Connection header received
0808 02:29:25.811056 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:25.811061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:25.811138 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:25.811254 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:25.812917 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.813671 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.815312 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.815774 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.815913 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:25.820132 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:25.820139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:25.820143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:25.820183 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:25.823736 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:25.824742 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:25.824750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:25.824755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:25.824929 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:25.825382 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:25.825388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:25.825393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:25.825798 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:25.825965 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:25.826750 (+ 785us) server_negotiation.cc:299] Negotiation successful
0808 02:29:25.827030 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4431,"thread_start_us":4341,"threads_started":1}
I20260808 02:29:25.828297 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:25.807097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:33557 (local address 127.19.231.2:59927)
0808 02:29:25.808034 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:25.808102 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:25.809992 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:25.811520 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:25.811529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:25.812136 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:25.812732 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:25.812749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.813825 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.813830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:25.815162 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:25.815175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:25.816008 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:25.816018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:25.816263 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:25.816987 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:25.817010 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:25.819820 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:25.823878 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:25.823890 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:25.823909 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:25.824590 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:25.825053 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:25.825057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:25.825060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:25.825254 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:25.825932 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:25.825937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:25.826293 (+ 356us) client_negotiation.cc:769] Sending connection context
0808 02:29:25.826525 (+ 232us) client_negotiation.cc:240] Negotiation successful
0808 02:29:25.827021 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":398,"threads_started":1}
I20260808 02:29:25.829903 7530 heartbeater.cc:347] Connected to a master server at 127.19.231.62:33557
I20260808 02:29:25.830240 7530 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:25.831033 7530 heartbeater.cc:511] Master 127.19.231.62:33557 requested a full tablet report, sending...
I20260808 02:29:25.832535 7174 ts_manager.cc:194] Registered new tserver with Master: 88b4eb73a33f4bfcb832227b8fd19183 (127.19.231.2:41197)
I20260808 02:29:25.833371 7174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:59927
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:29:25.881433 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33557
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:26.228926 7536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:26.229277 7536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:26.229465 7536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:26.237473 7536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:26.237624 7536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:26.237852 7536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:29:26.248860 7536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:26.250231 7536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:26.251722 7536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:26.265827 7544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:26.265892 7542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:26.266335 7541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:26.267390 7536 server_base.cc:1034] running on GCE node
I20260808 02:29:26.268038 7536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:26.269088 7536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:26.270370 7536 hybrid_clock.cc:648] HybridClock initialized: now 1786156166270291 us; error 63 us; skew 500 ppm
I20260808 02:29:26.270787 7536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:26 dist-test-slave-pgkr krb5kdc[7123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156166, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:26.274104 7536 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:29:26.275580 7536 webserver.cc:460] Webserver started at http://127.19.231.3:41551/ using document root <none> and password file <none>
I20260808 02:29:26.276257 7536 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:26.276362 7536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:26.276629 7536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:26.278463 7384 heartbeater.cc:503] Master 127.19.231.62:33557 was elected leader, sending a full tablet report...
I20260808 02:29:26.279791 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "ce74fe37126d43e5aa41d3bafa39d7c6"
format_stamp: "Formatted at 2026-08-08 02:29:26 on dist-test-slave-pgkr"
server_key: "ed80ea4f7e17f966f23cecf444508ba2"
server_key_iv: "20e19eb67f477b5e6a499fae85e3a528"
server_key_version: "encryptionkey@0"
I20260808 02:29:26.280614 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "ce74fe37126d43e5aa41d3bafa39d7c6"
format_stamp: "Formatted at 2026-08-08 02:29:26 on dist-test-slave-pgkr"
server_key: "ed80ea4f7e17f966f23cecf444508ba2"
server_key_iv: "20e19eb67f477b5e6a499fae85e3a528"
server_key_version: "encryptionkey@0"
I20260808 02:29:26.286161 7536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:29:26.289904 7553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:26.291354 7536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:29:26.291534 7536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "ce74fe37126d43e5aa41d3bafa39d7c6"
format_stamp: "Formatted at 2026-08-08 02:29:26 on dist-test-slave-pgkr"
server_key: "ed80ea4f7e17f966f23cecf444508ba2"
server_key_iv: "20e19eb67f477b5e6a499fae85e3a528"
server_key_version: "encryptionkey@0"
I20260808 02:29:26.291762 7536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:26.308426 7536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:26.312892 7536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:26.313197 7536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:26.314195 7536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:26.315613 7536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:26.315687 7536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:26.315860 7536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:26.315953 7536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:26.351445 7536 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:41599
I20260808 02:29:26.351545 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:41599 every 8 connection(s)
I20260808 02:29:26.352866 7536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:29:26.354974 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 7536
I20260808 02:29:26.355203 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:29:26.355751 20380 external_mini_cluster.cc:1428] Setting key c7aac065543dd34cd816c6de6e7aa188
Aug 08 02:29:26 dist-test-slave-pgkr krb5kdc[7123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156166, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:26.374389 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.355784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:37571 (local address 127.19.231.62:33557)
0808 02:29:26.356332 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.356337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.357554 (+ 1217us) server_negotiation.cc:407] Connection header received
0808 02:29:26.359592 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.359596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.359661 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.359976 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:26.361579 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.362401 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.363970 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.364376 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.364487 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.367955 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:26.367960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:26.367964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:26.367997 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:26.371210 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.371971 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.371976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.371978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.372100 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.372593 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.372597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.372599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.372864 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:26.373036 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.373766 (+ 730us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.373987 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":247,"threads_started":1}
I20260808 02:29:26.375547 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.356493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:33557 (local address 127.19.231.3:37571)
0808 02:29:26.357379 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.357426 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:26.359326 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:26.360257 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:26.360266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:26.360792 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:26.361399 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.361417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.362545 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.362549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.363824 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.363836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.364574 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.364578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.364854 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.365462 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:26.365483 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:26.367734 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:26.371363 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:26.371372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:26.371390 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:26.371831 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:26.372214 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:26.372220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:26.372223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:26.372448 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:26.373002 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:26.373010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:26.373376 (+ 366us) client_negotiation.cc:769] Sending connection context
0808 02:29:26.373647 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 02:29:26.374218 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":358,"threads_started":1}
I20260808 02:29:26.376976 7680 heartbeater.cc:347] Connected to a master server at 127.19.231.62:33557
I20260808 02:29:26.377308 7680 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:26.378324 7680 heartbeater.cc:511] Master 127.19.231.62:33557 requested a full tablet report, sending...
I20260808 02:29:26.379905 7174 ts_manager.cc:194] Registered new tserver with Master: ce74fe37126d43e5aa41d3bafa39d7c6 (127.19.231.3:41599)
I20260808 02:29:26.380738 7174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:37571
I20260808 02:29:26.383596 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:29:26.401602 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.387858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.19.231.62:33557)
0808 02:29:26.388081 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.388090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.388490 (+ 400us) server_negotiation.cc:407] Connection header received
0808 02:29:26.388648 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.388652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.388711 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.388792 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:26.390313 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.391089 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.392806 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.393251 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.393342 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.396220 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:26.396236 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:26.396239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:26.396269 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:26.398862 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.399541 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.399545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.399547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.399654 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.400082 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.400085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.400087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.400382 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:26.400568 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.401011 (+ 443us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.401243 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 02:29:26.406601 7174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:29:26.415108 7174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:29:26.450314 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.428143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.19.231.2:41197)
0808 02:29:26.437982 (+ 9839us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.437996 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.440339 (+ 2343us) server_negotiation.cc:407] Connection header received
0808 02:29:26.441246 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.441252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.441488 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.441682 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:26.443922 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.445554 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.447775 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.448712 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.448807 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.448960 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.449751 (+ 791us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.449958 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9581,"thread_start_us":9879,"threads_started":1}
I20260808 02:29:26.450326 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.427711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:36960)
0808 02:29:26.439017 (+ 11306us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.439076 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:26.439478 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:26.441974 (+ 2496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:26.441981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:26.442024 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:26.442735 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.442750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.444681 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.444693 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.447303 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.447326 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.449415 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.449422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.449646 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.449679 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 02:29:26.449750 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 02:29:26.449862 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10952,"mutex_wait_us":77,"spinlock_wait_cycles":1792,"thread_start_us":12130,"threads_started":1}
I20260808 02:29:26.450603 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.428060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.19.231.1:42871)
0808 02:29:26.441054 (+ 12994us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.441062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.441090 (+ 28us) server_negotiation.cc:407] Connection header received
0808 02:29:26.441198 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.441206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.441490 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.441689 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:26.442950 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.444537 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.447520 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.449083 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.449231 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.449459 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.450074 (+ 615us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.450257 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12816,"thread_start_us":12644,"threads_started":1}
I20260808 02:29:26.450991 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.427912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41197 (local address 127.0.0.1:57120)
0808 02:29:26.440188 (+ 12276us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.440211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:26.441045 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:26.441984 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:26.441988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:26.442030 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:26.443660 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.443679 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.445700 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.445708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.447552 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.447568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.449117 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.449121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.449427 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.449494 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 02:29:26.449639 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 02:29:26.449861 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12115,"mutex_wait_us":442,"spinlock_wait_cycles":3328,"thread_start_us":10557,"threads_started":1}
I20260808 02:29:26.453485 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.440963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:41599 (local address 127.0.0.1:54944)
0808 02:29:26.442645 (+ 1682us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.442666 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:26.442802 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:26.445096 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:26.445101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:26.445124 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:26.445563 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.445575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.447599 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.447606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.449826 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.449840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.452675 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.452681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.452851 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.452887 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:29:26.452992 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 02:29:26.453129 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1505,"mutex_wait_us":48,"thread_start_us":1302,"threads_started":1}
I20260808 02:29:26.453569 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.440713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.19.231.3:41599)
0808 02:29:26.444546 (+ 3833us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.444554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.444588 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:29:26.444701 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.444708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.444895 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.445100 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:26.445784 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.447430 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.450010 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.452143 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.452298 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.452517 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.453072 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.453258 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3654,"thread_start_us":3493,"threads_started":1}
I20260808 02:29:26.455026 7314 tablet_service.cc:1467] Processing CreateTablet for tablet d920751556304b2fa089038868bcbf04 (DEFAULT_TABLE table=test-table [id=cbcc5a10a68e45a0b693ee4752682111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:26.455061 7462 tablet_service.cc:1467] Processing CreateTablet for tablet d920751556304b2fa089038868bcbf04 (DEFAULT_TABLE table=test-table [id=cbcc5a10a68e45a0b693ee4752682111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:26.456341 7314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d920751556304b2fa089038868bcbf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:26.456445 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d920751556304b2fa089038868bcbf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:26.457384 7610 tablet_service.cc:1467] Processing CreateTablet for tablet d920751556304b2fa089038868bcbf04 (DEFAULT_TABLE table=test-table [id=cbcc5a10a68e45a0b693ee4752682111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:26.458771 7610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d920751556304b2fa089038868bcbf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:26.468130 7700 tablet_bootstrap.cc:492] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: Bootstrap starting.
I20260808 02:29:26.471796 7700 tablet_bootstrap.cc:654] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:26.471987 7701 tablet_bootstrap.cc:492] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Bootstrap starting.
I20260808 02:29:26.473985 7700 log.cc:824] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:26.474568 7703 tablet_bootstrap.cc:492] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Bootstrap starting.
I20260808 02:29:26.475909 7701 tablet_bootstrap.cc:654] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:26.477408 7703 tablet_bootstrap.cc:654] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:26.477437 7701 log.cc:824] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:26.478564 7703 log.cc:824] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:26.481093 7703 tablet_bootstrap.cc:492] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: No bootstrap required, opened a new log
I20260808 02:29:26.481281 7701 tablet_bootstrap.cc:492] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: No bootstrap required, opened a new log
I20260808 02:29:26.481400 7703 ts_tablet_manager.cc:1397] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 02:29:26.481560 7701 ts_tablet_manager.cc:1397] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 02:29:26.484134 7700 tablet_bootstrap.cc:492] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: No bootstrap required, opened a new log
I20260808 02:29:26.484429 7700 ts_tablet_manager.cc:1397] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260808 02:29:26.489660 7703 raft_consensus.cc:348] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.490067 7703 raft_consensus.cc:374] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:26.490173 7703 raft_consensus.cc:729] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce74fe37126d43e5aa41d3bafa39d7c6, State: Initialized, Role: FOLLOWER
I20260808 02:29:26.490854 7703 consensus_queue.cc:260] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.492035 7680 heartbeater.cc:503] Master 127.19.231.62:33557 was elected leader, sending a full tablet report...
I20260808 02:29:26.492789 7703 ts_tablet_manager.cc:1428] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260808 02:29:26.494234 7701 raft_consensus.cc:348] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.494714 7701 raft_consensus.cc:374] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:26.494827 7701 raft_consensus.cc:729] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b4eb73a33f4bfcb832227b8fd19183, State: Initialized, Role: FOLLOWER
I20260808 02:29:26.495311 7700 raft_consensus.cc:348] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.495520 7701 consensus_queue.cc:260] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.495761 7700 raft_consensus.cc:374] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:26.495864 7700 raft_consensus.cc:729] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24c8a682c734674b03db1a51dc5d4b9, State: Initialized, Role: FOLLOWER
I20260808 02:29:26.496824 7530 heartbeater.cc:503] Master 127.19.231.62:33557 was elected leader, sending a full tablet report...
I20260808 02:29:26.496732 7700 consensus_queue.cc:260] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.496999 7707 raft_consensus.cc:482] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:26.497288 7707 raft_consensus.cc:504] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.499574 7707 leader_election.cc:288] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a24c8a682c734674b03db1a51dc5d4b9 (127.19.231.1:42871), 88b4eb73a33f4bfcb832227b8fd19183 (127.19.231.2:41197)
I20260808 02:29:26.500806 7701 ts_tablet_manager.cc:1428] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20260808 02:29:26.503481 7700 ts_tablet_manager.cc:1428] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: Time spent starting tablet: real 0.019s user 0.013s sys 0.003s
I20260808 02:29:26.508446 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.500439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:35699 (local address 127.19.231.2:41197)
0808 02:29:26.500640 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.500647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.500667 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:29:26.500751 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.500760 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.500866 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.500962 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:26.501714 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.503097 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.506112 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.507444 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.507579 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.507656 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.508150 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.508262 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:29:26.508528 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.499870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41197 (local address 127.19.231.3:35699)
0808 02:29:26.500331 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.500353 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:26.500474 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:26.501194 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:26.501202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:26.501239 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:26.501509 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.501524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.503319 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.503328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.505909 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.505933 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.507745 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.507750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.507977 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.508013 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 02:29:26.508121 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 02:29:26.508266 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260808 02:29:26.509429 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d920751556304b2fa089038868bcbf04" candidate_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4eb73a33f4bfcb832227b8fd19183" is_pre_election: true
I20260808 02:29:26.509907 7485 raft_consensus.cc:2454] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce74fe37126d43e5aa41d3bafa39d7c6 in term 0.
I20260808 02:29:26.510598 7555 leader_election.cc:302] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b4eb73a33f4bfcb832227b8fd19183, ce74fe37126d43e5aa41d3bafa39d7c6; no voters:
I20260808 02:29:26.511049 7707 raft_consensus.cc:2781] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:29:26.511210 7707 raft_consensus.cc:482] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:29:26.511318 7707 raft_consensus.cc:3037] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:26.512887 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.500512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:49463 (local address 127.19.231.1:42871)
0808 02:29:26.500697 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.500703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.507073 (+ 6370us) server_negotiation.cc:407] Connection header received
0808 02:29:26.507281 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.507286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.507361 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.507459 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:26.507972 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.509014 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.510788 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.511980 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.512146 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.512196 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.512618 (+ 422us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.512709 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 02:29:26.513084 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.500044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.19.231.3:49463)
0808 02:29:26.506975 (+ 6931us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.506995 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:26.507129 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:26.507481 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:26.507485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:26.507506 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:26.507818 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.507825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.509166 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.509170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.510592 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:26.510604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.512208 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.512215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:26.512445 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.512479 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 02:29:26.512582 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 02:29:26.512719 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6780,"thread_start_us":6604,"threads_started":1}
I20260808 02:29:26.514088 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d920751556304b2fa089038868bcbf04" candidate_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24c8a682c734674b03db1a51dc5d4b9" is_pre_election: true
I20260808 02:29:26.514431 7707 raft_consensus.cc:504] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.514585 7337 raft_consensus.cc:2454] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce74fe37126d43e5aa41d3bafa39d7c6 in term 0.
I20260808 02:29:26.515233 7707 leader_election.cc:288] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [CANDIDATE]: Term 1 election: Requested vote from peers a24c8a682c734674b03db1a51dc5d4b9 (127.19.231.1:42871), 88b4eb73a33f4bfcb832227b8fd19183 (127.19.231.2:41197)
I20260808 02:29:26.515568 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d920751556304b2fa089038868bcbf04" candidate_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24c8a682c734674b03db1a51dc5d4b9"
I20260808 02:29:26.515779 7337 raft_consensus.cc:3037] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:26.515755 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d920751556304b2fa089038868bcbf04" candidate_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b4eb73a33f4bfcb832227b8fd19183"
I20260808 02:29:26.515960 7485 raft_consensus.cc:3037] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:26.517988 7337 raft_consensus.cc:2454] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce74fe37126d43e5aa41d3bafa39d7c6 in term 1.
I20260808 02:29:26.518213 7485 raft_consensus.cc:2454] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce74fe37126d43e5aa41d3bafa39d7c6 in term 1.
I20260808 02:29:26.518563 7557 leader_election.cc:302] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a24c8a682c734674b03db1a51dc5d4b9, ce74fe37126d43e5aa41d3bafa39d7c6; no voters:
I20260808 02:29:26.518961 7707 raft_consensus.cc:2781] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:26.519455 7707 raft_consensus.cc:686] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 1 LEADER]: Becoming Leader. State: Replica: ce74fe37126d43e5aa41d3bafa39d7c6, State: Running, Role: LEADER
I20260808 02:29:26.519944 7707 consensus_queue.cc:237] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } }
I20260808 02:29:26.524712 7173 catalog_manager.cc:5495] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce74fe37126d43e5aa41d3bafa39d7c6 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce74fe37126d43e5aa41d3bafa39d7c6" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41599 } health_report { overall_health: HEALTHY } } }
I20260808 02:29:26.555064 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.549555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.19.231.3:41599)
0808 02:29:26.549799 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.549805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.549828 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:29:26.549910 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.549914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.549988 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.550144 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:29:26.550564 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.551364 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.552776 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.553176 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.553313 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.553918 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:29:26.554415 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:29:26.554615 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.554797 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.554848 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 02:29:26.555720 7531 tablet.cc:2369] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:26.564432 7337 raft_consensus.cc:1261] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 1 FOLLOWER]: Refusing update from remote peer ce74fe37126d43e5aa41d3bafa39d7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:26.564432 7485 raft_consensus.cc:1261] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 1 FOLLOWER]: Refusing update from remote peer ce74fe37126d43e5aa41d3bafa39d7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:26.565542 7711 consensus_queue.cc:1035] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b4eb73a33f4bfcb832227b8fd19183" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:26.565958 7707 consensus_queue.cc:1035] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24c8a682c734674b03db1a51dc5d4b9" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 42871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:26.596556 7174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:26.596892 7174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:26.600577 7174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a693df407e84129a31bd2c33b3a9ba6: Sending DeleteTablet for 3 replicas of tablet d920751556304b2fa089038868bcbf04
I20260808 02:29:26.601889 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet d920751556304b2fa089038868bcbf04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:26 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:57120
I20260808 02:29:26.601918 7314 tablet_service.cc:1514] Processing DeleteTablet for tablet d920751556304b2fa089038868bcbf04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:26 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:36960
I20260808 02:29:26.601922 7610 tablet_service.cc:1514] Processing DeleteTablet for tablet d920751556304b2fa089038868bcbf04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:26 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:54944
I20260808 02:29:26.606139 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 7242
I20260808 02:29:26.620028 7726 tablet_replica.cc:330] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: stopping tablet replica
I20260808 02:29:26.620553 7726 raft_consensus.cc:2229] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:29:26.621388 7726 raft_consensus.cc:2258] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:26.622170 7728 tablet_replica.cc:330] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: stopping tablet replica
I20260808 02:29:26.622649 7728 raft_consensus.cc:2229] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:26.623183 7728 raft_consensus.cc:2258] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:26.624243 7726 ts_tablet_manager.cc:1905] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:26.627214 7727 tablet_replica.cc:330] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: stopping tablet replica
I20260808 02:29:26.627696 7727 raft_consensus.cc:2229] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:26.628242 7727 raft_consensus.cc:2258] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 08 02:29:26 dist-test-slave-pgkr krb5kdc[7123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:29:26.630676 7728 ts_tablet_manager.cc:1905] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:26.633908 7726 ts_tablet_manager.cc:1918] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:26.634155 7726 log.cc:1196] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/d920751556304b2fa089038868bcbf04
I20260808 02:29:26.634863 7726 ts_tablet_manager.cc:1939] T d920751556304b2fa089038868bcbf04 P ce74fe37126d43e5aa41d3bafa39d7c6: Deleting consensus metadata
I20260808 02:29:26.636909 7159 catalog_manager.cc:4841] TS ce74fe37126d43e5aa41d3bafa39d7c6 (127.19.231.3:41599): tablet d920751556304b2fa089038868bcbf04 (table test-table [id=cbcc5a10a68e45a0b693ee4752682111]) successfully deleted
I20260808 02:29:26.638176 7727 ts_tablet_manager.cc:1905] T d920751556304b2fa089038868bcbf04 P a24c8a682c734674b03db1a51dc5d4b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:26.639345 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.607710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.19.231.1:42871)
0808 02:29:26.607953 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.607962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.610904 (+ 2942us) server_negotiation.cc:407] Connection header received
0808 02:29:26.611013 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.611022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.611148 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.611295 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:26.614975 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.616250 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.623656 (+ 7406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.624384 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.624498 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.630723 (+ 6225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:26.630739 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:26.630757 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:26.630810 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:26.634900 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.635900 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.635910 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.635914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.636091 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.637182 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.637188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.637194 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.637657 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:26.637950 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.638671 (+ 721us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.638932 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:29:26.639766 7728 ts_tablet_manager.cc:1918] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:26.640003 7728 log.cc:1196] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/d920751556304b2fa089038868bcbf04
I20260808 02:29:26.640620 7728 ts_tablet_manager.cc:1939] T d920751556304b2fa089038868bcbf04 P 88b4eb73a33f4bfcb832227b8fd19183: Deleting consensus metadata
I20260808 02:29:26.640872 7379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:26.642586 7157 catalog_manager.cc:4841] TS 88b4eb73a33f4bfcb832227b8fd19183 (127.19.231.2:41197): tablet d920751556304b2fa089038868bcbf04 (table test-table [id=cbcc5a10a68e45a0b693ee4752682111]) successfully deleted
I20260808 02:29:26.733088 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 7242
I20260808 02:29:26.758230 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 7390
W20260808 02:29:26.758230 7159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.231.1:42871 (error 108)
I20260808 02:29:26.760795 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.760243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:36980)
0808 02:29:26.760494 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.760619 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260808 02:29:26.761086 7159 catalog_manager.cc:4584] TS a24c8a682c734674b03db1a51dc5d4b9 (127.19.231.1:42871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d920751556304b2fa089038868bcbf04: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Aug 08 02:29:26 dist-test-slave-pgkr krb5kdc[7123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:29:26.776288 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.759389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.19.231.2:41197)
0808 02:29:26.759589 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.759595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.759799 (+ 204us) server_negotiation.cc:407] Connection header received
0808 02:29:26.760057 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.760065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.760154 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.760290 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:26.761961 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.762847 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.764262 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.764706 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.764869 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.769870 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:26.769886 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:26.769902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:26.769944 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:26.773167 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.773964 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.773980 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.773983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.774130 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.774523 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.774542 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.774545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.775035 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:26.775356 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.775671 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.775933 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:29:26.776948 7525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:26.815173 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.814513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:36990)
0808 02:29:26.814801 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.814935 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260808 02:29:26.816438 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.816023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:37002)
0808 02:29:26.816191 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.816295 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 02:29:26.873667 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 7390
I20260808 02:29:26.878826 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.878146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:37010)
0808 02:29:26.878398 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.878571 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 02:29:26.880024 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.879641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:37018)
0808 02:29:26.879808 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.879897 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 02:29:26.905294 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 7536
Aug 08 02:29:26 dist-test-slave-pgkr krb5kdc[7123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:29:26.924809 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.906271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.19.231.3:41599)
0808 02:29:26.906545 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:26.906551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:26.906572 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:29:26.906760 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:26.906767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:26.906853 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:26.906983 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:26.908358 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.909642 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.911011 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:26.911803 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:26.911959 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:26.917087 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:26.917110 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:26.917130 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:26.917181 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:26.921345 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.922118 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.922128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.922132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.922363 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:26.923048 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:26.923053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:26.923056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:26.923550 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:26.923885 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:26.924114 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 02:29:26.924336 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260808 02:29:26.925680 7675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:26.960880 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.960344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:37032)
0808 02:29:26.960575 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.960692 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:29:26.961998 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:26.961563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:37040)
0808 02:29:26.961773 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:26.961864 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260808 02:29:27.052430 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 7536
I20260808 02:29:27.079104 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 7137
I20260808 02:29:27.080224 7208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:27.193782 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:27.192969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:42871 (local address 127.0.0.1:37048)
0808 02:29:27.193287 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:27.193425 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:42871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 02:29:27.193908 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 7137
2026-08-08T02:29:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4671 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T02:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:27Z Disabled control of system clock
I20260808 02:29:27.298386 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43033
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:43033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:29:27.664865 7739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:27.665169 7739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:27.665257 7739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:27.673243 7739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:27.673369 7739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:27.673416 7739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:27.673475 7739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:27.673533 7739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:27.682947 7739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:43033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:43033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:27.684229 7739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:27.685985 7739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:27.697912 7745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:27.698019 7747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:27.697993 7744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:27.698961 7739 server_base.cc:1034] running on GCE node
I20260808 02:29:27.699798 7739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:27.701181 7739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:27.702415 7739 hybrid_clock.cc:648] HybridClock initialized: now 1786156167702373 us; error 54 us; skew 500 ppm
I20260808 02:29:27.702823 7739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:27.705081 7739 webserver.cc:460] Webserver started at http://127.19.231.62:36383/ using document root <none> and password file <none>
I20260808 02:29:27.705776 7739 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:27.705914 7739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:27.706248 7739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:27.709259 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "3f57d2900c7b41909fa8320bfb033e99"
format_stamp: "Formatted at 2026-08-08 02:29:27 on dist-test-slave-pgkr"
server_key: "a8304c271a3da40765becdae25638ad3"
server_key_iv: "764e65cb9f7888ad8c39de8a23640db7"
server_key_version: "encryptionkey@0"
I20260808 02:29:27.710145 7739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "3f57d2900c7b41909fa8320bfb033e99"
format_stamp: "Formatted at 2026-08-08 02:29:27 on dist-test-slave-pgkr"
server_key: "a8304c271a3da40765becdae25638ad3"
server_key_iv: "764e65cb9f7888ad8c39de8a23640db7"
server_key_version: "encryptionkey@0"
I20260808 02:29:27.715319 7739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:29:27.718880 7753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:27.720227 7739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:29:27.720418 7739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "3f57d2900c7b41909fa8320bfb033e99"
format_stamp: "Formatted at 2026-08-08 02:29:27 on dist-test-slave-pgkr"
server_key: "a8304c271a3da40765becdae25638ad3"
server_key_iv: "764e65cb9f7888ad8c39de8a23640db7"
server_key_version: "encryptionkey@0"
I20260808 02:29:27.720633 7739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:27.744899 7739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:27.745837 7739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:27.746170 7739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:27.766722 7739 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:43033
I20260808 02:29:27.766686 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:43033 every 8 connection(s)
I20260808 02:29:27.768306 7739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:27.770931 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 7739
I20260808 02:29:27.771160 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:27.771698 20380 external_mini_cluster.cc:1428] Setting key 821a660d30178e2d4f94e7840f49a0f9
I20260808 02:29:27.772459 7811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:27.786985 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:27.773675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.19.231.62:43033)
0808 02:29:27.774809 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:27.774825 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:27.774882 (+ 57us) server_negotiation.cc:407] Connection header received
0808 02:29:27.776814 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:27.776842 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:27.777467 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:27.777896 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:27.778440 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:27.780601 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:27.781983 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:27.783004 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:27.783223 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:27.784535 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:27.784544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:27.784558 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:27.785163 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:27.785243 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:27.785264 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:27.785396 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:27.785640 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 02:29:27.785961 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260808 02:29:27.787263 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99: Bootstrap starting.
I20260808 02:29:27.791153 7811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:27.792182 7811 log.cc:824] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:27.795008 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99: No bootstrap required, opened a new log
I20260808 02:29:27.802877 7811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f57d2900c7b41909fa8320bfb033e99" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43033 } }
I20260808 02:29:27.803365 7811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:27.803498 7811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f57d2900c7b41909fa8320bfb033e99, State: Initialized, Role: FOLLOWER
I20260808 02:29:27.804181 7811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f57d2900c7b41909fa8320bfb033e99" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43033 } }
I20260808 02:29:27.804390 7811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:27.804493 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:27.804627 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:27.806795 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f57d2900c7b41909fa8320bfb033e99" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43033 } }
I20260808 02:29:27.807250 7811 leader_election.cc:302] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f57d2900c7b41909fa8320bfb033e99; no voters:
I20260808 02:29:27.807857 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:27.808013 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:27.808766 7817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [term 1 LEADER]: Becoming Leader. State: Replica: 3f57d2900c7b41909fa8320bfb033e99, State: Running, Role: LEADER
I20260808 02:29:27.809370 7817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f57d2900c7b41909fa8320bfb033e99" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43033 } }
I20260808 02:29:27.809998 7811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:27.812296 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f57d2900c7b41909fa8320bfb033e99. Latest consensus state: current_term: 1 leader_uuid: "3f57d2900c7b41909fa8320bfb033e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f57d2900c7b41909fa8320bfb033e99" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43033 } } }
I20260808 02:29:27.812860 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:27.812633 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f57d2900c7b41909fa8320bfb033e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f57d2900c7b41909fa8320bfb033e99" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 43033 } } }
I20260808 02:29:27.813067 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:27.815232 7823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:27.820731 7823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:27.830883 7823 catalog_manager.cc:1347] Generated new cluster ID: 121cb0e477ff44e69557fc7d72f5966f
I20260808 02:29:27.831024 7823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:27.842504 7823 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:27.844221 7823 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:27.855553 7823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99: Generated new TSK 0
I20260808 02:29:27.856454 7823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:27.868909 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43033
--builtin_ntp_servers=127.19.231.20:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:29:28.246206 7836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:28.246479 7836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:28.246574 7836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:28.253901 7836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:28.253999 7836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:28.254104 7836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:29:28.263119 7836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:28.264360 7836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:28.266089 7836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:28.279652 7842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:28.280061 7843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:28.280316 7845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:28.281574 7836 server_base.cc:1034] running on GCE node
I20260808 02:29:28.282322 7836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:28.283517 7836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:28.284797 7836 hybrid_clock.cc:648] HybridClock initialized: now 1786156168284687 us; error 91 us; skew 500 ppm
I20260808 02:29:28.285162 7836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:28.287882 7836 webserver.cc:460] Webserver started at http://127.19.231.1:42145/ using document root <none> and password file <none>
I20260808 02:29:28.288669 7836 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:28.288841 7836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:28.289258 7836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:28.292665 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "56d377ec7a234090adebaddb24f5ee09"
format_stamp: "Formatted at 2026-08-08 02:29:28 on dist-test-slave-pgkr"
server_key: "82e697e6ecc7a7ec6b2ca4efe54b6ed2"
server_key_iv: "5c9dd8e1f86c3585f46ccdfef74fa64b"
server_key_version: "encryptionkey@0"
I20260808 02:29:28.293648 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "56d377ec7a234090adebaddb24f5ee09"
format_stamp: "Formatted at 2026-08-08 02:29:28 on dist-test-slave-pgkr"
server_key: "82e697e6ecc7a7ec6b2ca4efe54b6ed2"
server_key_iv: "5c9dd8e1f86c3585f46ccdfef74fa64b"
server_key_version: "encryptionkey@0"
I20260808 02:29:28.299834 7836 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 02:29:28.303648 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:28.304981 7836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:29:28.305178 7836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "56d377ec7a234090adebaddb24f5ee09"
format_stamp: "Formatted at 2026-08-08 02:29:28 on dist-test-slave-pgkr"
server_key: "82e697e6ecc7a7ec6b2ca4efe54b6ed2"
server_key_iv: "5c9dd8e1f86c3585f46ccdfef74fa64b"
server_key_version: "encryptionkey@0"
I20260808 02:29:28.305420 7836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:28.336293 7836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:28.337217 7836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:28.337561 7836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:28.338672 7836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:28.340288 7836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:28.340428 7836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:28.340569 7836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:28.340668 7836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:28.377004 7836 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:38163
I20260808 02:29:28.377058 7971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:38163 every 8 connection(s)
I20260808 02:29:28.378439 7836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:29:28.379817 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 7836
I20260808 02:29:28.380052 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:29:28.380544 20380 external_mini_cluster.cc:1428] Setting key a8ccbdccc6ed8dc641068ec5cf6144f8
I20260808 02:29:28.383728 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43033
--builtin_ntp_servers=127.19.231.20:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:29:28.393265 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:28.381463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:36469 (local address 127.19.231.62:43033)
0808 02:29:28.382025 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:28.382033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:28.383530 (+ 1497us) server_negotiation.cc:407] Connection header received
0808 02:29:28.385601 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:28.385609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:28.385698 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:28.385820 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:28.387035 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.388146 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.389849 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.390545 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.390733 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:28.392213 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:28.392222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:28.392229 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:28.392254 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:28.392321 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:28.392340 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:28.392436 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:28.392713 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 02:29:28.392822 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":281,"threads_started":1}
I20260808 02:29:28.394129 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:28.382183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43033 (local address 127.19.231.1:36469)
0808 02:29:28.383350 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:28.383399 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:28.385296 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:28.386024 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:28.386033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:28.386150 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:28.386853 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:28.386874 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.388319 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.388323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:28.389697 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:28.389711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.390823 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.390828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:28.391064 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:28.391858 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:28.391861 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:28.391902 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:28.391987 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:28.392469 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:28.392479 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:28.392486 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:29:28.392670 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 02:29:28.392963 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":391,"threads_started":1}
I20260808 02:29:28.395788 7972 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43033
I20260808 02:29:28.396135 7972 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:28.397043 7972 heartbeater.cc:511] Master 127.19.231.62:43033 requested a full tablet report, sending...
I20260808 02:29:28.399668 7771 ts_manager.cc:194] Registered new tserver with Master: 56d377ec7a234090adebaddb24f5ee09 (127.19.231.1:38163)
I20260808 02:29:28.401299 7771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.1:36469
W20260808 02:29:28.737993 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:28.738360 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:28.738513 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:28.745755 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:28.745888 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:28.746073 7976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:29:28.755416 7976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:28.756769 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:28.758749 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:28.772603 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:28.772893 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:28.772768 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:28.780315 7976 server_base.cc:1034] running on GCE node
I20260808 02:29:28.781015 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:28.782006 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:28.783242 7976 hybrid_clock.cc:648] HybridClock initialized: now 1786156168783245 us; error 106 us; skew 500 ppm
I20260808 02:29:28.783568 7976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:28.785873 7976 webserver.cc:460] Webserver started at http://127.19.231.2:38849/ using document root <none> and password file <none>
I20260808 02:29:28.786571 7976 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:28.786764 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:28.787103 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:28.789981 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb"
format_stamp: "Formatted at 2026-08-08 02:29:28 on dist-test-slave-pgkr"
server_key: "87031d9920e64ec4e7a2fdc04864c9d4"
server_key_iv: "d2cb6116023938707c34eb97b4b00d96"
server_key_version: "encryptionkey@0"
I20260808 02:29:28.790858 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb"
format_stamp: "Formatted at 2026-08-08 02:29:28 on dist-test-slave-pgkr"
server_key: "87031d9920e64ec4e7a2fdc04864c9d4"
server_key_iv: "d2cb6116023938707c34eb97b4b00d96"
server_key_version: "encryptionkey@0"
I20260808 02:29:28.796003 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 02:29:28.799574 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:28.800993 7976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:29:28.801129 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb"
format_stamp: "Formatted at 2026-08-08 02:29:28 on dist-test-slave-pgkr"
server_key: "87031d9920e64ec4e7a2fdc04864c9d4"
server_key_iv: "d2cb6116023938707c34eb97b4b00d96"
server_key_version: "encryptionkey@0"
I20260808 02:29:28.801309 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:28.822898 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:28.823671 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:28.824034 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:28.825176 7976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:28.826776 7976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:28.826867 7976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:28.826989 7976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:28.827064 7976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:28.864149 7976 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:33169
I20260808 02:29:28.864307 8115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:33169 every 8 connection(s)
I20260808 02:29:28.865484 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:29:28.870676 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 7976
I20260808 02:29:28.870885 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:29:28.871512 20380 external_mini_cluster.cc:1428] Setting key ad2937b30acc64eecd88d7ea624ee3fe
I20260808 02:29:28.875063 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43033
--builtin_ntp_servers=127.19.231.20:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:29:28.881356 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:28.868138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:44609 (local address 127.19.231.62:43033)
0808 02:29:28.868381 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:28.868387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:28.870104 (+ 1717us) server_negotiation.cc:407] Connection header received
0808 02:29:28.872905 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:28.872909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:28.872974 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:28.873057 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:28.874509 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.875371 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.878210 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.878739 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.878842 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:28.880605 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:28.880610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:28.880613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:28.880631 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:28.880661 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:28.880673 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:28.880745 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:28.881035 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 02:29:28.881121 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:29:28.883091 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:28.868783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43033 (local address 127.19.231.2:44609)
0808 02:29:28.869902 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:28.869986 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:28.872569 (+ 2583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:28.873351 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:28.873363 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:28.873500 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:28.874331 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:28.874358 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.875543 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.875562 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:28.878031 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:28.878051 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:28.879001 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:28.879012 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:28.879387 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:28.880211 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:28.880218 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:28.880277 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:28.880405 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:28.880777 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:28.880793 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:28.880806 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 02:29:28.881013 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 02:29:28.881472 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":324,"threads_started":1}
I20260808 02:29:28.884562 8116 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43033
I20260808 02:29:28.884922 8116 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:28.885977 8116 heartbeater.cc:511] Master 127.19.231.62:43033 requested a full tablet report, sending...
I20260808 02:29:28.887784 7771 ts_manager.cc:194] Registered new tserver with Master: f49f4b5ac3044500bd5dc86fc8b7fccb (127.19.231.2:33169)
I20260808 02:29:28.888886 7771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.2:44609
W20260808 02:29:29.221103 8119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:29.221379 8119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:29.221531 8119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:29.228809 8119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:29.228901 8119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:29.229071 8119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:29:29.237993 8119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:43033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:29.239399 8119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:29.241231 8119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:29.256667 8125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:29.256927 8124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:29.261333 8127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:29.261945 8119 server_base.cc:1034] running on GCE node
I20260808 02:29:29.262594 8119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:29.263626 8119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:29.264868 8119 hybrid_clock.cc:648] HybridClock initialized: now 1786156169264825 us; error 38 us; skew 500 ppm
I20260808 02:29:29.265204 8119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:29.271507 8119 webserver.cc:460] Webserver started at http://127.19.231.3:39259/ using document root <none> and password file <none>
I20260808 02:29:29.272279 8119 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:29.272415 8119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:29.272747 8119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:29.275863 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "4bbfde8508b148d6bea60ed63e22a504"
format_stamp: "Formatted at 2026-08-08 02:29:29 on dist-test-slave-pgkr"
server_key: "aea6307ba029a3fa22c1abf878486bcf"
server_key_iv: "335535abca2827c045a8d6cac77bab0b"
server_key_version: "encryptionkey@0"
I20260808 02:29:29.276680 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "4bbfde8508b148d6bea60ed63e22a504"
format_stamp: "Formatted at 2026-08-08 02:29:29 on dist-test-slave-pgkr"
server_key: "aea6307ba029a3fa22c1abf878486bcf"
server_key_iv: "335535abca2827c045a8d6cac77bab0b"
server_key_version: "encryptionkey@0"
I20260808 02:29:29.281986 8119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:29:29.285755 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:29.287292 8119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:29.287467 8119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "4bbfde8508b148d6bea60ed63e22a504"
format_stamp: "Formatted at 2026-08-08 02:29:29 on dist-test-slave-pgkr"
server_key: "aea6307ba029a3fa22c1abf878486bcf"
server_key_iv: "335535abca2827c045a8d6cac77bab0b"
server_key_version: "encryptionkey@0"
I20260808 02:29:29.287693 8119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:29.306155 8119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:29.307149 8119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:29.307451 8119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:29.308460 8119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:29.310091 8119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:29.310217 8119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:29.310405 8119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:29.310490 8119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:29.347060 8119 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:41735
I20260808 02:29:29.347150 8254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:41735 every 8 connection(s)
I20260808 02:29:29.348438 8119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:29:29.351836 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 8119
I20260808 02:29:29.352041 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:29:29.352658 20380 external_mini_cluster.cc:1428] Setting key 848c1a518a0389d008eb81d2526241e5
I20260808 02:29:29.361838 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.351259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:37569 (local address 127.19.231.62:43033)
0808 02:29:29.351456 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.351461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.353414 (+ 1953us) server_negotiation.cc:407] Connection header received
0808 02:29:29.355389 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.355395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.355485 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.355602 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:29.356669 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.357412 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.359009 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.359439 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.359571 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.360964 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:29.360993 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:29.360997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:29.361013 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:29.361051 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:29.361062 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:29.361191 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.361412 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.361509 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:29:29.362905 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.351872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:43033 (local address 127.19.231.3:37569)
0808 02:29:29.352998 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.353044 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:29.355132 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:29.355830 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:29.355837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:29.355920 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:29.356498 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.356514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.357569 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.357573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.358866 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.358878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.359676 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.359685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.359929 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.360647 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:29.360650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:29.360687 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:29.360758 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:29.361157 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:29.361173 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:29.361181 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 02:29:29.361402 (+ 221us) client_negotiation.cc:240] Negotiation successful
0808 02:29:29.361790 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260808 02:29:29.364423 8255 heartbeater.cc:347] Connected to a master server at 127.19.231.62:43033
I20260808 02:29:29.364770 8255 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:29.365561 8255 heartbeater.cc:511] Master 127.19.231.62:43033 requested a full tablet report, sending...
I20260808 02:29:29.367120 7771 ts_manager.cc:194] Registered new tserver with Master: 4bbfde8508b148d6bea60ed63e22a504 (127.19.231.3:41735)
I20260808 02:29:29.368144 7771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.3:37569
I20260808 02:29:29.368299 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:29:29.379889 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.373459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.19.231.62:43033)
0808 02:29:29.373707 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.373717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.374154 (+ 437us) server_negotiation.cc:407] Connection header received
0808 02:29:29.374342 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.374347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.374407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.374492 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:29.375089 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.375819 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.377185 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.377854 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.378002 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.379104 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:29.379110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:29.379115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:29.379136 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:29.379176 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:29.379191 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:29.379345 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.379515 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.379576 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:29:29.383877 7771 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:29:29.390813 7771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:29:29.405018 7972 heartbeater.cc:503] Master 127.19.231.62:43033 was elected leader, sending a full tablet report...
I20260808 02:29:29.439688 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.403506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32916)
0808 02:29:29.430147 (+ 26641us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.430164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:29.430286 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:29.431932 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:29.431936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:29.431965 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:29.432447 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.432470 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.434544 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.434548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.436925 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.436941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.438922 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.438928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.439100 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.439131 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:29:29.439235 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 02:29:29.439351 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26541,"mutex_wait_us":190,"thread_start_us":23127,"threads_started":1}
I20260808 02:29:29.439884 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.404180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.19.231.1:38163)
0808 02:29:29.425621 (+ 21441us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.425636 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.430405 (+ 4769us) server_negotiation.cc:407] Connection header received
0808 02:29:29.430538 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.430545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.431451 (+ 906us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.431655 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:29.432656 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.434326 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.437124 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.438709 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.438857 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.439139 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.439296 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.439474 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21099,"thread_start_us":20922,"threads_started":1}
I20260808 02:29:29.440357 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.402971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:41735 (local address 127.0.0.1:43320)
0808 02:29:29.426191 (+ 23220us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.426262 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:29.428891 (+ 2629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:29.430805 (+ 1914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:29.430812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:29.430841 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:29.431410 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.431426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.433983 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.433991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.437127 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.437141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.439282 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.439290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.439478 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.439509 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 02:29:29.439609 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 02:29:29.439750 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22825,"mutex_wait_us":431,"thread_start_us":27630,"threads_started":1}
I20260808 02:29:29.440754 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.403491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:33169 (local address 127.0.0.1:46382)
0808 02:29:29.429580 (+ 26089us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.429603 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:29.429722 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:29.431476 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:29.431481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:29.431503 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:29.431999 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.432015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.434472 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.434481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.437057 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.437075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.438422 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.438429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.438644 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.438716 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 02:29:29.438877 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 02:29:29.439094 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25955,"mutex_wait_us":9,"thread_start_us":25798,"threads_started":1}
I20260808 02:29:29.441402 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.403350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.19.231.3:41735)
0808 02:29:29.430040 (+ 26690us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.430048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.430085 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:29:29.430194 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.430202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.430457 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.432495 (+ 2038us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:29.432817 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.433843 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.437326 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.438736 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.438857 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.439139 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.440803 (+ 1664us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.440983 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26478,"thread_start_us":26300,"threads_started":1}
I20260808 02:29:29.441653 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.404169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.19.231.2:33169)
0808 02:29:29.429873 (+ 25704us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.429884 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.429921 (+ 37us) server_negotiation.cc:407] Connection header received
0808 02:29:29.430102 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.430114 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.430436 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.432108 (+ 1672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:29.432764 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.434326 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.437265 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.438214 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.438372 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.438671 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.441120 (+ 2449us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.441301 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25536,"thread_start_us":26970,"threads_started":1}
I20260808 02:29:29.444042 7902 tablet_service.cc:1467] Processing CreateTablet for tablet 3798bdcd773b41c5be3d6c04cec187e9 (DEFAULT_TABLE table=test-table [id=118993ba51a24397932d15e760e63f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:29.444552 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 3798bdcd773b41c5be3d6c04cec187e9 (DEFAULT_TABLE table=test-table [id=118993ba51a24397932d15e760e63f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:29.445621 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3798bdcd773b41c5be3d6c04cec187e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:29.446029 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3798bdcd773b41c5be3d6c04cec187e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:29.447252 8045 tablet_service.cc:1467] Processing CreateTablet for tablet 3798bdcd773b41c5be3d6c04cec187e9 (DEFAULT_TABLE table=test-table [id=118993ba51a24397932d15e760e63f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:29.448588 8045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3798bdcd773b41c5be3d6c04cec187e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:29.457772 8275 tablet_bootstrap.cc:492] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Bootstrap starting.
I20260808 02:29:29.461716 8275 tablet_bootstrap.cc:654] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:29.462836 8275 log.cc:824] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:29.463548 8276 tablet_bootstrap.cc:492] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Bootstrap starting.
I20260808 02:29:29.465448 8277 tablet_bootstrap.cc:492] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Bootstrap starting.
I20260808 02:29:29.467090 8275 tablet_bootstrap.cc:492] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: No bootstrap required, opened a new log
I20260808 02:29:29.467372 8275 ts_tablet_manager.cc:1397] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 02:29:29.467705 8276 tablet_bootstrap.cc:654] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:29.469004 8277 tablet_bootstrap.cc:654] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:29.469008 8276 log.cc:824] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:29.470156 8277 log.cc:824] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:29.472220 8276 tablet_bootstrap.cc:492] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: No bootstrap required, opened a new log
I20260808 02:29:29.472543 8276 ts_tablet_manager.cc:1397] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 02:29:29.473338 8277 tablet_bootstrap.cc:492] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: No bootstrap required, opened a new log
I20260808 02:29:29.473594 8277 ts_tablet_manager.cc:1397] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260808 02:29:29.476693 8275 raft_consensus.cc:348] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.477104 8275 raft_consensus.cc:374] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:29.477242 8275 raft_consensus.cc:729] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbfde8508b148d6bea60ed63e22a504, State: Initialized, Role: FOLLOWER
I20260808 02:29:29.477788 8275 consensus_queue.cc:260] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.480644 8255 heartbeater.cc:503] Master 127.19.231.62:43033 was elected leader, sending a full tablet report...
I20260808 02:29:29.481202 8275 ts_tablet_manager.cc:1428] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260808 02:29:29.483304 8276 raft_consensus.cc:348] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.483747 8276 raft_consensus.cc:374] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:29.483856 8276 raft_consensus.cc:729] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d377ec7a234090adebaddb24f5ee09, State: Initialized, Role: FOLLOWER
I20260808 02:29:29.484472 8276 consensus_queue.cc:260] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.486267 8276 ts_tablet_manager.cc:1428] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260808 02:29:29.485961 8277 raft_consensus.cc:348] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.486421 8277 raft_consensus.cc:374] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:29.486527 8277 raft_consensus.cc:729] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49f4b5ac3044500bd5dc86fc8b7fccb, State: Initialized, Role: FOLLOWER
I20260808 02:29:29.487272 8277 consensus_queue.cc:260] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.488524 8116 heartbeater.cc:503] Master 127.19.231.62:43033 was elected leader, sending a full tablet report...
I20260808 02:29:29.489221 8277 ts_tablet_manager.cc:1428] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260808 02:29:29.600214 8256 tablet.cc:2369] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:29:29.617795 8117 tablet.cc:2369] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 02:29:29.631026 7973 tablet.cc:2369] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:29.657823 8281 raft_consensus.cc:482] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:29.658171 8281 raft_consensus.cc:504] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.659930 8281 leader_election.cc:288] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f49f4b5ac3044500bd5dc86fc8b7fccb (127.19.231.2:33169), 56d377ec7a234090adebaddb24f5ee09 (127.19.231.1:38163)
I20260808 02:29:29.666872 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.660500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.19.231.3:35141)
0808 02:29:29.660687 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.660712 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:29.660884 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:29.661384 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:29.661392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:29.661424 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:29.661737 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.661749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.662942 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.662946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.664801 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.664815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.666092 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.666100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.666260 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.666299 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:29:29.666397 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 02:29:29.666537 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 02:29:29.666882 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.660446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:35141 (local address 127.19.231.1:38163)
0808 02:29:29.660661 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.660667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.660822 (+ 155us) server_negotiation.cc:407] Connection header received
0808 02:29:29.661058 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.661067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.661183 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.661307 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:29.661901 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.662814 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.664961 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.665899 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.666056 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.666124 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.666443 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.666573 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:29:29.667717 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.660446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:49053 (local address 127.19.231.2:33169)
0808 02:29:29.660660 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.660667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.661713 (+ 1046us) server_negotiation.cc:407] Connection header received
0808 02:29:29.661903 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.661908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.662003 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.662101 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:29.662746 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.663504 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.665585 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.666678 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.666840 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.666916 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.667403 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.667542 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:29:29.667881 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.660507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:33169 (local address 127.19.231.3:49053)
0808 02:29:29.661584 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.661605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:29.661743 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:29.662155 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:29.662159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:29.662182 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:29.662493 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.662500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.663685 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.663691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.665401 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:29.665412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.667037 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.667042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:29.667219 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.667252 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 02:29:29.667354 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 02:29:29.667486 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":178,"thread_start_us":432,"threads_started":1}
I20260808 02:29:29.668332 7925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3798bdcd773b41c5be3d6c04cec187e9" candidate_uuid: "4bbfde8508b148d6bea60ed63e22a504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d377ec7a234090adebaddb24f5ee09" is_pre_election: true
I20260808 02:29:29.668865 7925 raft_consensus.cc:2454] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbfde8508b148d6bea60ed63e22a504 in term 0.
I20260808 02:29:29.668859 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3798bdcd773b41c5be3d6c04cec187e9" candidate_uuid: "4bbfde8508b148d6bea60ed63e22a504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" is_pre_election: true
I20260808 02:29:29.669443 8067 raft_consensus.cc:2454] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbfde8508b148d6bea60ed63e22a504 in term 0.
I20260808 02:29:29.669755 8136 leader_election.cc:302] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbfde8508b148d6bea60ed63e22a504, 56d377ec7a234090adebaddb24f5ee09; no voters:
I20260808 02:29:29.670156 8281 raft_consensus.cc:2781] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:29:29.670302 8281 raft_consensus.cc:482] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:29:29.670429 8281 raft_consensus.cc:3037] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:29.672950 8281 raft_consensus.cc:504] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.673588 8281 leader_election.cc:288] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [CANDIDATE]: Term 1 election: Requested vote from peers f49f4b5ac3044500bd5dc86fc8b7fccb (127.19.231.2:33169), 56d377ec7a234090adebaddb24f5ee09 (127.19.231.1:38163)
I20260808 02:29:29.674131 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3798bdcd773b41c5be3d6c04cec187e9" candidate_uuid: "4bbfde8508b148d6bea60ed63e22a504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb"
I20260808 02:29:29.674232 7925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3798bdcd773b41c5be3d6c04cec187e9" candidate_uuid: "4bbfde8508b148d6bea60ed63e22a504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d377ec7a234090adebaddb24f5ee09"
I20260808 02:29:29.674355 8067 raft_consensus.cc:3037] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:29.674496 7925 raft_consensus.cc:3037] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:29.676889 8067 raft_consensus.cc:2454] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbfde8508b148d6bea60ed63e22a504 in term 1.
I20260808 02:29:29.676892 7925 raft_consensus.cc:2454] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbfde8508b148d6bea60ed63e22a504 in term 1.
I20260808 02:29:29.677320 8137 leader_election.cc:302] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbfde8508b148d6bea60ed63e22a504, f49f4b5ac3044500bd5dc86fc8b7fccb; no voters:
I20260808 02:29:29.677608 8281 raft_consensus.cc:2781] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:29.678046 8281 raft_consensus.cc:686] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbfde8508b148d6bea60ed63e22a504, State: Running, Role: LEADER
I20260808 02:29:29.678524 8281 consensus_queue.cc:237] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } }
I20260808 02:29:29.683095 7770 catalog_manager.cc:5495] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbfde8508b148d6bea60ed63e22a504 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "4bbfde8508b148d6bea60ed63e22a504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbfde8508b148d6bea60ed63e22a504" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 } health_report { overall_health: UNKNOWN } } }
I20260808 02:29:29.736063 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.731101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.19.231.3:41735)
0808 02:29:29.731292 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.731296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.731315 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:29:29.731402 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.731405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.731491 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.731620 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:29:29.731977 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.732709 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.734021 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.734438 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.734528 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.735045 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:29:29.735572 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:29:29.735704 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.735836 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.735873 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 02:29:29.744956 8067 raft_consensus.cc:1261] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 1 FOLLOWER]: Refusing update from remote peer 4bbfde8508b148d6bea60ed63e22a504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:29.744949 7925 raft_consensus.cc:1261] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 1 FOLLOWER]: Refusing update from remote peer 4bbfde8508b148d6bea60ed63e22a504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:29.745965 8285 consensus_queue.cc:1035] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49f4b5ac3044500bd5dc86fc8b7fccb" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 33169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:29.746376 8281 consensus_queue.cc:1035] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d377ec7a234090adebaddb24f5ee09" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 38163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:29.758292 8289 mvcc.cc:205] Tried to move back new op lower bound from 7316095671263637504 to 7316095671007121408. Current Snapshot: MvccSnapshot[applied={T|T < 7316095671263637504}]
I20260808 02:29:29.763799 8291 mvcc.cc:205] Tried to move back new op lower bound from 7316095671263637504 to 7316095671007121408. Current Snapshot: MvccSnapshot[applied={T|T < 7316095671263637504}]
I20260808 02:29:29.775569 7770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:29.775862 7770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:29.779480 7770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f57d2900c7b41909fa8320bfb033e99: Sending DeleteTablet for 3 replicas of tablet 3798bdcd773b41c5be3d6c04cec187e9
I20260808 02:29:29.780602 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 3798bdcd773b41c5be3d6c04cec187e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:29 UTC) from {username='slave'} at 127.0.0.1:43320
I20260808 02:29:29.780674 8045 tablet_service.cc:1514] Processing DeleteTablet for tablet 3798bdcd773b41c5be3d6c04cec187e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:29 UTC) from {username='slave'} at 127.0.0.1:46382
I20260808 02:29:29.781126 7902 tablet_service.cc:1514] Processing DeleteTablet for tablet 3798bdcd773b41c5be3d6c04cec187e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:29 UTC) from {username='slave'} at 127.0.0.1:32916
I20260808 02:29:29.783439 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 7836
I20260808 02:29:29.792150 8304 tablet_replica.cc:330] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: stopping tablet replica
I20260808 02:29:29.792811 8304 raft_consensus.cc:2229] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:29.793500 8304 raft_consensus.cc:2258] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:29.796977 8304 ts_tablet_manager.cc:1905] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:29.799576 8300 tablet_replica.cc:330] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: stopping tablet replica
I20260808 02:29:29.799978 8300 raft_consensus.cc:2229] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:29:29.800571 8300 raft_consensus.cc:2258] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:29.801280 8303 tablet_replica.cc:330] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: stopping tablet replica
I20260808 02:29:29.801638 8303 raft_consensus.cc:2229] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:29.802071 8303 raft_consensus.cc:2258] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:29.806210 8300 ts_tablet_manager.cc:1905] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:29.806388 8303 ts_tablet_manager.cc:1905] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:29.807682 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.784791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.19.231.1:38163)
0808 02:29:29.784992 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.785000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.790428 (+ 5428us) server_negotiation.cc:407] Connection header received
0808 02:29:29.790678 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.790684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.790793 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.790884 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:29.794300 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.795582 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.802849 (+ 7267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.803567 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.803695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.805506 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:29.805519 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:29.805552 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:29.806360 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:29.806427 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:29.806464 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:29.806579 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.806927 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.807185 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:29:29.809101 8304 ts_tablet_manager.cc:1918] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:29.809338 8304 log.cc:1196] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/3798bdcd773b41c5be3d6c04cec187e9
I20260808 02:29:29.809376 7967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:29.822770 8300 ts_tablet_manager.cc:1918] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:29.823143 8300 log.cc:1196] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/3798bdcd773b41c5be3d6c04cec187e9
I20260808 02:29:29.823884 8300 ts_tablet_manager.cc:1939] T 3798bdcd773b41c5be3d6c04cec187e9 P 4bbfde8508b148d6bea60ed63e22a504: Deleting consensus metadata
I20260808 02:29:29.824046 8303 ts_tablet_manager.cc:1918] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:29.824314 8303 log.cc:1196] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/3798bdcd773b41c5be3d6c04cec187e9
I20260808 02:29:29.824987 8303 ts_tablet_manager.cc:1939] T 3798bdcd773b41c5be3d6c04cec187e9 P f49f4b5ac3044500bd5dc86fc8b7fccb: Deleting consensus metadata
I20260808 02:29:29.826401 7755 catalog_manager.cc:4841] TS 4bbfde8508b148d6bea60ed63e22a504 (127.19.231.3:41735): tablet 3798bdcd773b41c5be3d6c04cec187e9 (table test-table [id=118993ba51a24397932d15e760e63f22]) successfully deleted
I20260808 02:29:29.826997 7757 catalog_manager.cc:4841] TS f49f4b5ac3044500bd5dc86fc8b7fccb (127.19.231.2:33169): tablet 3798bdcd773b41c5be3d6c04cec187e9 (table test-table [id=118993ba51a24397932d15e760e63f22]) successfully deleted
I20260808 02:29:29.912407 8304 ts_tablet_manager.cc:1939] T 3798bdcd773b41c5be3d6c04cec187e9 P 56d377ec7a234090adebaddb24f5ee09: Deleting consensus metadata
I20260808 02:29:29.912535 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 7836
I20260808 02:29:29.937004 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 7976
W20260808 02:29:29.937142 7756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.231.1:38163 (error 108)
I20260808 02:29:29.939906 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.939375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32936)
0808 02:29:29.939586 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.939719 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 02:29:29.940307 7756 catalog_manager.cc:4584] TS 56d377ec7a234090adebaddb24f5ee09 (127.19.231.1:38163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3798bdcd773b41c5be3d6c04cec187e9: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
I20260808 02:29:29.945068 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.937933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.19.231.2:33169)
0808 02:29:29.938142 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:29.938149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:29.938170 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:29:29.938360 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:29.938367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:29.938498 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:29.938669 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:29.939183 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.940304 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.941754 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:29.942257 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:29.942390 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:29.943671 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:29.943681 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:29.943696 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:29.944295 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:29.944343 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:29.944364 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:29.944500 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:29.944623 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 02:29:29.944741 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:29:29.945753 8111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:29.994002 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.993407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32940)
0808 02:29:29.993689 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.993819 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 02:29:29.995253 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:29.994826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32944)
0808 02:29:29.995004 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:29.995102 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 02:29:30.047025 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 7976
I20260808 02:29:30.056763 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:30.056165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32950)
0808 02:29:30.056424 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:30.056536 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 02:29:30.058063 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:30.057594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32954)
0808 02:29:30.057793 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:30.057891 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 02:29:30.071666 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 8119
I20260808 02:29:30.080158 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:30.072740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.19.231.3:41735)
0808 02:29:30.072964 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:30.072970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:30.072994 (+ 24us) server_negotiation.cc:407] Connection header received
0808 02:29:30.073156 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:30.073162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:30.073277 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:30.073387 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:30.073787 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:30.075016 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:30.076392 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:30.077068 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:30.077205 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:30.078278 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:30.078290 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:30.078311 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:30.079351 (+ 1040us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:30.079422 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:30.079449 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:30.079606 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:30.079664 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 02:29:30.079763 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 02:29:30.080946 8250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:30.138867 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:30.138248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32964)
0808 02:29:30.138507 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:30.138652 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 02:29:30.140162 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:30.139697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32968)
0808 02:29:30.139920 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:30.140009 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260808 02:29:30.205847 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 8119
I20260808 02:29:30.230340 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 7739
I20260808 02:29:30.231542 7805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:30.345350 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:30.344755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:38163 (local address 127.0.0.1:32980)
0808 02:29:30.344979 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:30.345110 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:38163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 02:29:30.345508 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 7739
2026-08-08T02:29:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3150 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:30 dist-test-slave-pgkr krb5kdc[8313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:30 dist-test-slave-pgkr krb5kdc[8313](info): set up 2 sockets
Aug 08 02:29:30 dist-test-slave-pgkr krb5kdc[8313](info): commencing operation
krb5kdc: starting...
W20260808 02:29:31.535357 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:31 dist-test-slave-pgkr krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:29:31.870393 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40119
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:40119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:32.225056 8326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:32.225353 8326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:32.225533 8326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:32.232790 8326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:32.232929 8326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:32.233022 8326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:32.233139 8326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:32.233207 8326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:32.243291 8326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:40119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:32.244836 8326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:32.246197 8326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:32.258348 8331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:32.258419 8332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:32.258445 8334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:32.259522 8326 server_base.cc:1034] running on GCE node
I20260808 02:29:32.260257 8326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:32.261699 8326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:32.262892 8326 hybrid_clock.cc:648] HybridClock initialized: now 1786156172262832 us; error 52 us; skew 500 ppm
I20260808 02:29:32.263273 8326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:32 dist-test-slave-pgkr krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156172, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:32.266463 8326 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:29:32.268040 8326 webserver.cc:460] Webserver started at http://127.19.231.62:43221/ using document root <none> and password file <none>
I20260808 02:29:32.268729 8326 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:32.268833 8326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:32.269089 8326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:32.272275 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "b62d70387505411584f963dc5408e80d"
format_stamp: "Formatted at 2026-08-08 02:29:32 on dist-test-slave-pgkr"
server_key: "5a6d88a5cec85a18cdb50ac5af1c6ad5"
server_key_iv: "9071ffb3ed50439155386b1297c6ec1c"
server_key_version: "encryptionkey@0"
I20260808 02:29:32.273033 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "b62d70387505411584f963dc5408e80d"
format_stamp: "Formatted at 2026-08-08 02:29:32 on dist-test-slave-pgkr"
server_key: "5a6d88a5cec85a18cdb50ac5af1c6ad5"
server_key_iv: "9071ffb3ed50439155386b1297c6ec1c"
server_key_version: "encryptionkey@0"
I20260808 02:29:32.278774 8326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:29:32.282435 8343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:32.283991 8326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:32.284189 8326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "b62d70387505411584f963dc5408e80d"
format_stamp: "Formatted at 2026-08-08 02:29:32 on dist-test-slave-pgkr"
server_key: "5a6d88a5cec85a18cdb50ac5af1c6ad5"
server_key_iv: "9071ffb3ed50439155386b1297c6ec1c"
server_key_version: "encryptionkey@0"
I20260808 02:29:32.284407 8326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:32.313331 8326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:32.317497 8326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:32.317800 8326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:32.337385 8326 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:40119
I20260808 02:29:32.337433 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:40119 every 8 connection(s)
I20260808 02:29:32.338833 8326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:32.342986 8397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:32.344532 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 8326
I20260808 02:29:32.344758 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:32.345414 20380 external_mini_cluster.cc:1428] Setting key 7047a28fe4e27032e79f20ef853640ff
I20260808 02:29:32.359175 8397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d: Bootstrap starting.
Aug 08 02:29:32 dist-test-slave-pgkr krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:32.363268 8397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:32.364516 8397 log.cc:824] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:32.367851 8397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d: No bootstrap required, opened a new log
I20260808 02:29:32.370229 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:32.347485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.19.231.62:40119)
0808 02:29:32.348318 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:32.348331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:32.348372 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:29:32.350031 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:32.350058 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:32.350454 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:32.350854 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:32.352899 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:32.354169 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:32.356340 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:32.357011 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:32.357137 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:32.362471 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:32.362480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:32.362495 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:32.362535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:32.365949 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:32.366795 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:32.366802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:32.366808 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:32.366975 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:32.367401 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:32.367405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:32.367407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:32.367931 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:32.368220 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:32.368753 (+ 533us) server_negotiation.cc:299] Negotiation successful
0808 02:29:32.369097 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":389,"threads_started":1}
I20260808 02:29:32.377667 8397 raft_consensus.cc:348] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62d70387505411584f963dc5408e80d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40119 } }
I20260808 02:29:32.378266 8397 raft_consensus.cc:374] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:32.378400 8397 raft_consensus.cc:729] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62d70387505411584f963dc5408e80d, State: Initialized, Role: FOLLOWER
I20260808 02:29:32.379161 8397 consensus_queue.cc:260] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62d70387505411584f963dc5408e80d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40119 } }
I20260808 02:29:32.379496 8397 raft_consensus.cc:388] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:32.379683 8397 raft_consensus.cc:482] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:32.379932 8397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:32.382515 8397 raft_consensus.cc:504] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62d70387505411584f963dc5408e80d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40119 } }
I20260808 02:29:32.383200 8397 leader_election.cc:302] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b62d70387505411584f963dc5408e80d; no voters:
I20260808 02:29:32.383847 8397 leader_election.cc:288] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:32.384017 8402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:32.384763 8402 raft_consensus.cc:686] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [term 1 LEADER]: Becoming Leader. State: Replica: b62d70387505411584f963dc5408e80d, State: Running, Role: LEADER
I20260808 02:29:32.385464 8402 consensus_queue.cc:237] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62d70387505411584f963dc5408e80d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40119 } }
I20260808 02:29:32.386266 8397 sys_catalog.cc:564] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:32.388319 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b62d70387505411584f963dc5408e80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62d70387505411584f963dc5408e80d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40119 } } }
I20260808 02:29:32.388666 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:32.389487 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b62d70387505411584f963dc5408e80d. Latest consensus state: current_term: 1 leader_uuid: "b62d70387505411584f963dc5408e80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62d70387505411584f963dc5408e80d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40119 } } }
I20260808 02:29:32.389712 8407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:32.392724 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:32.396356 8407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:32.405778 8407 catalog_manager.cc:1347] Generated new cluster ID: 713d5b8b15c84fe8b1052d4b37b22121
I20260808 02:29:32.405895 8407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:32.427657 8407 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:32.429829 8407 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:32.441293 8407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d: Generated new TSK 0
I20260808 02:29:32.442654 8407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.231.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
Entry for principal kudu/127.19.231.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab.
I20260808 02:29:32.545894 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40119
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:32.924180 8424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:32.924512 8424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:32.924695 8424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:32.931923 8424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:32.932073 8424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:32.932278 8424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:29:32.942438 8424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.1.keytab
--principal=kudu/127.19.231.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:32.943792 8424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:32.945122 8424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:32.958777 8431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:32.958886 8429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:32.958811 8434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:32.964097 8424 server_base.cc:1034] running on GCE node
I20260808 02:29:32.964710 8424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:32.965620 8424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:32.966830 8424 hybrid_clock.cc:648] HybridClock initialized: now 1786156172966771 us; error 59 us; skew 500 ppm
I20260808 02:29:32.967180 8424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:32 dist-test-slave-pgkr krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156172, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:32.970373 8424 init.cc:377] Logged in from keytab as kudu/127.19.231.1@KRBTEST.COM (short username kudu)
I20260808 02:29:32.972046 8424 webserver.cc:460] Webserver started at http://127.19.231.1:34957/ using document root <none> and password file <none>
I20260808 02:29:32.972803 8424 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:32.972954 8424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:32.973290 8424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:32.976337 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "45a85c470aaa497ea3289ad17ee7ea6c"
format_stamp: "Formatted at 2026-08-08 02:29:32 on dist-test-slave-pgkr"
server_key: "a42e912caf0bbd07ce7b66fb7848f767"
server_key_iv: "40da894b891091c671ff1a47e58d3ffc"
server_key_version: "encryptionkey@0"
I20260808 02:29:32.977202 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "45a85c470aaa497ea3289ad17ee7ea6c"
format_stamp: "Formatted at 2026-08-08 02:29:32 on dist-test-slave-pgkr"
server_key: "a42e912caf0bbd07ce7b66fb7848f767"
server_key_iv: "40da894b891091c671ff1a47e58d3ffc"
server_key_version: "encryptionkey@0"
I20260808 02:29:32.982569 8424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:29:32.986138 8441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:32.987520 8424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:32.987705 8424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "45a85c470aaa497ea3289ad17ee7ea6c"
format_stamp: "Formatted at 2026-08-08 02:29:32 on dist-test-slave-pgkr"
server_key: "a42e912caf0bbd07ce7b66fb7848f767"
server_key_iv: "40da894b891091c671ff1a47e58d3ffc"
server_key_version: "encryptionkey@0"
I20260808 02:29:32.987941 8424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:33.011353 8424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:33.015743 8424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:33.016062 8424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:33.017105 8424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:33.018553 8424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:33.018678 8424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 02:29:33.018839 8424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:33.018915 8424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:33.054625 8424 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:41055
I20260808 02:29:33.054821 8563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:41055 every 8 connection(s)
I20260808 02:29:33.056016 8424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:29:33.059873 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 8424
I20260808 02:29:33.060124 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:29:33.060756 20380 external_mini_cluster.cc:1428] Setting key 8e04bb068521972de4514cd15262dd4d
Aug 08 02:29:33 dist-test-slave-pgkr krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156172, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.1@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:33.078848 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:33.058782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:34617 (local address 127.19.231.62:40119)
0808 02:29:33.059306 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:33.059312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:33.060438 (+ 1126us) server_negotiation.cc:407] Connection header received
0808 02:29:33.062892 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:33.062896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:33.062955 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:33.063039 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:33.065246 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.065969 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.068496 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.068905 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.069056 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:33.072930 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:33.072935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:33.072939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:33.072970 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:33.075652 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:33.076430 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:33.076434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:33.076436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:33.076564 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:33.077027 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:33.077031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:33.077032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:33.077316 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:33.077480 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:33.078201 (+ 721us) server_negotiation.cc:299] Negotiation successful
0808 02:29:33.078424 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
I20260808 02:29:33.079960 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:33.059340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40119 (local address 127.19.231.1:34617)
0808 02:29:33.060234 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:33.060301 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:33.062521 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:33.063335 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:33.063346 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:33.064167 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:33.065044 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:33.065083 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.066132 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.066141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:33.068340 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:33.068359 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.069153 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.069163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:33.069417 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:33.070124 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:33.070146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:33.072694 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:33.075814 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:33.075823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:33.075840 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:33.076287 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:33.076683 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:33.076688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:33.076692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:33.076893 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:33.077459 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:33.077467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:33.077816 (+ 349us) client_negotiation.cc:769] Sending connection context
0808 02:29:33.078099 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 02:29:33.078696 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":323,"threads_started":1}
I20260808 02:29:33.081655 8564 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40119
I20260808 02:29:33.081988 8564 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:33.082789 8564 heartbeater.cc:511] Master 127.19.231.62:40119 requested a full tablet report, sending...
I20260808 02:29:33.085182 8361 ts_manager.cc:194] Registered new tserver with Master: 45a85c470aaa497ea3289ad17ee7ea6c (127.19.231.1:41055)
I20260808 02:29:33.087680 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.1@KRBTEST.COM'} at 127.19.231.1:34617
WARNING: no policy specified for kudu/127.19.231.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
Entry for principal kudu/127.19.231.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab.
I20260808 02:29:33.137552 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40119
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:33.490275 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:33.490557 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:33.490718 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:33.498277 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:33.498399 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:33.498512 8573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:29:33.508831 8573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.2.keytab
--principal=kudu/127.19.231.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:33.510102 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:33.511547 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:33.525518 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:33.525483 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:33.525413 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:33.526255 8573 server_base.cc:1034] running on GCE node
I20260808 02:29:33.526944 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:33.527937 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:33.529234 8573 hybrid_clock.cc:648] HybridClock initialized: now 1786156173529177 us; error 31 us; skew 500 ppm
I20260808 02:29:33.529632 8573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:33 dist-test-slave-pgkr krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156173, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:33.533128 8573 init.cc:377] Logged in from keytab as kudu/127.19.231.2@KRBTEST.COM (short username kudu)
I20260808 02:29:33.534945 8573 webserver.cc:460] Webserver started at http://127.19.231.2:38321/ using document root <none> and password file <none>
I20260808 02:29:33.535730 8573 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:33.535897 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:33.536268 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:33.539574 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "522ed0719a184154b3d3b46dc40b010c"
format_stamp: "Formatted at 2026-08-08 02:29:33 on dist-test-slave-pgkr"
server_key: "8fa26102f7197cc61a215b21a31d1bed"
server_key_iv: "997a98392184ea726de99bb65b11a76a"
server_key_version: "encryptionkey@0"
I20260808 02:29:33.540416 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "522ed0719a184154b3d3b46dc40b010c"
format_stamp: "Formatted at 2026-08-08 02:29:33 on dist-test-slave-pgkr"
server_key: "8fa26102f7197cc61a215b21a31d1bed"
server_key_iv: "997a98392184ea726de99bb65b11a76a"
server_key_version: "encryptionkey@0"
I20260808 02:29:33.546154 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 02:29:33.550037 8590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:33.551669 8573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:33.551936 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "522ed0719a184154b3d3b46dc40b010c"
format_stamp: "Formatted at 2026-08-08 02:29:33 on dist-test-slave-pgkr"
server_key: "8fa26102f7197cc61a215b21a31d1bed"
server_key_iv: "997a98392184ea726de99bb65b11a76a"
server_key_version: "encryptionkey@0"
I20260808 02:29:33.552182 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:33.565735 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:33.570711 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:33.571058 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:33.572101 8573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:33.573643 8573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:33.573756 8573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:33.573923 8573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:33.574007 8573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:33.613741 8573 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:41573
I20260808 02:29:33.613790 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:41573 every 8 connection(s)
I20260808 02:29:33.615228 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:29:33.621769 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 8573
I20260808 02:29:33.621989 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:29:33.622450 20380 external_mini_cluster.cc:1428] Setting key a5884b28dd3356ec300b710b893731c7
Aug 08 02:29:33 dist-test-slave-pgkr krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156173, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.2@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:33.636283 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:33.618067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:33235 (local address 127.19.231.62:40119)
0808 02:29:33.618549 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:33.618554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:33.619538 (+ 984us) server_negotiation.cc:407] Connection header received
0808 02:29:33.621425 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:33.621430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:33.621518 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:33.621639 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:33.623520 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.624255 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.625809 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.626224 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.626363 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:33.629953 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:33.629958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:33.629961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:33.629992 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:33.633090 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:33.633828 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:33.633834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:33.633838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:33.633977 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:33.634428 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:33.634433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:33.634436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:33.634779 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:33.634938 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:33.635646 (+ 708us) server_negotiation.cc:299] Negotiation successful
0808 02:29:33.635886 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":269,"threads_started":1}
I20260808 02:29:33.637390 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:33.618631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40119 (local address 127.19.231.2:33235)
0808 02:29:33.619361 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:33.619408 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:33.621122 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:33.621935 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:33.621944 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:33.622547 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:33.623346 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:33.623368 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.624416 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.624421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:33.625655 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:33.625668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:33.626456 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:33.626462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:33.626813 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:33.627520 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:33.627543 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:33.629731 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:33.633240 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:33.633250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:33.633270 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:33.633685 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:33.634096 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:33.634101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:33.634111 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:33.634301 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:33.634913 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:33.634921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:33.635266 (+ 345us) client_negotiation.cc:769] Sending connection context
0808 02:29:33.635548 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 02:29:33.636134 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":281,"threads_started":1}
I20260808 02:29:33.638962 8708 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40119
I20260808 02:29:33.639294 8708 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:33.640044 8708 heartbeater.cc:511] Master 127.19.231.62:40119 requested a full tablet report, sending...
I20260808 02:29:33.641572 8361 ts_manager.cc:194] Registered new tserver with Master: 522ed0719a184154b3d3b46dc40b010c (127.19.231.2:41573)
I20260808 02:29:33.642411 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.2@KRBTEST.COM'} at 127.19.231.2:33235
WARNING: no policy specified for kudu/127.19.231.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
Entry for principal kudu/127.19.231.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab.
I20260808 02:29:33.697975 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40119
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:34.068320 8714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:34.068605 8714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:34.068765 8714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:34.076377 8714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:34.076489 8714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:34.076730 8714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:29:34.087046 8714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.3.keytab
--principal=kudu/127.19.231.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:34.088596 8714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:34.090032 8714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 02:29:34.090873 8564 heartbeater.cc:503] Master 127.19.231.62:40119 was elected leader, sending a full tablet report...
W20260808 02:29:34.104455 8721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:34.104429 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:34.104429 8720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:34.105630 8714 server_base.cc:1034] running on GCE node
I20260808 02:29:34.106197 8714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:34.107195 8714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:34.108426 8714 hybrid_clock.cc:648] HybridClock initialized: now 1786156174108370 us; error 39 us; skew 500 ppm
I20260808 02:29:34.108816 8714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:34 dist-test-slave-pgkr krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156174, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:34.112152 8714 init.cc:377] Logged in from keytab as kudu/127.19.231.3@KRBTEST.COM (short username kudu)
I20260808 02:29:34.113705 8714 webserver.cc:460] Webserver started at http://127.19.231.3:36599/ using document root <none> and password file <none>
I20260808 02:29:34.114418 8714 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:34.114579 8714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:34.114939 8714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:34.118021 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "2ee7aab7699a4b85ac82b5da3d50cf61"
format_stamp: "Formatted at 2026-08-08 02:29:34 on dist-test-slave-pgkr"
server_key: "3034dbf56617a84ec004475322162667"
server_key_iv: "2d355dbbfc99009d433b0c0a32d4aa80"
server_key_version: "encryptionkey@0"
I20260808 02:29:34.118849 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "2ee7aab7699a4b85ac82b5da3d50cf61"
format_stamp: "Formatted at 2026-08-08 02:29:34 on dist-test-slave-pgkr"
server_key: "3034dbf56617a84ec004475322162667"
server_key_iv: "2d355dbbfc99009d433b0c0a32d4aa80"
server_key_version: "encryptionkey@0"
I20260808 02:29:34.124045 8714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:29:34.127599 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:34.128890 8714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 02:29:34.129078 8714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "2ee7aab7699a4b85ac82b5da3d50cf61"
format_stamp: "Formatted at 2026-08-08 02:29:34 on dist-test-slave-pgkr"
server_key: "3034dbf56617a84ec004475322162667"
server_key_iv: "2d355dbbfc99009d433b0c0a32d4aa80"
server_key_version: "encryptionkey@0"
I20260808 02:29:34.129292 8714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:34.150218 8714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:34.154914 8714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:34.155316 8714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:34.156629 8714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:34.158149 8714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:34.158231 8714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:34.158318 8714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:34.158365 8714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:34.192655 8714 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:42775
I20260808 02:29:34.192723 8849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:42775 every 8 connection(s)
I20260808 02:29:34.194144 8714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:29:34.202999 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 8714
I20260808 02:29:34.203239 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:29:34.203702 20380 external_mini_cluster.cc:1428] Setting key 1a1ef1df4c3d8264ea2e6d79083c0c4d
Aug 08 02:29:34 dist-test-slave-pgkr krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156174, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.3@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:29:34.215332 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.196933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:52077 (local address 127.19.231.62:40119)
0808 02:29:34.197483 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.197490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.198504 (+ 1014us) server_negotiation.cc:407] Connection header received
0808 02:29:34.200394 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.200399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.200485 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.200633 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:34.202172 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.203038 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.204924 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.205348 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.205490 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.209229 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:34.209234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:34.209237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:34.209269 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:34.212102 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.212850 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.212856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.212859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.212993 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.213454 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.213465 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.213468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.213747 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:34.213929 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.214680 (+ 751us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.214911 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":257,"threads_started":1}
I20260808 02:29:34.216328 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.197577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40119 (local address 127.19.231.3:52077)
0808 02:29:34.198330 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.198371 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.200108 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.200877 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.200884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.201404 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:34.201994 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.202012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.203195 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.203202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.204768 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.204782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.205593 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.205601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.205877 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.206682 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:29:34.206714 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:34.208991 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:34.212248 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:34.212269 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:34.212286 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:34.212725 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:34.213114 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:29:34.213120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:29:34.213124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:29:34.213332 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:29:34.213882 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:34.213891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:34.214278 (+ 387us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.214557 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.215171 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":283,"threads_started":1}
I20260808 02:29:34.217823 8850 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40119
I20260808 02:29:34.218149 8850 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:34.218935 8850 heartbeater.cc:511] Master 127.19.231.62:40119 requested a full tablet report, sending...
I20260808 02:29:34.220395 8361 ts_manager.cc:194] Registered new tserver with Master: 2ee7aab7699a4b85ac82b5da3d50cf61 (127.19.231.3:42775)
I20260808 02:29:34.221217 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.231.3@KRBTEST.COM'} at 127.19.231.3:52077
I20260808 02:29:34.231251 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:29:34.249037 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.235792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.19.231.62:40119)
0808 02:29:34.236028 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.236036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.236347 (+ 311us) server_negotiation.cc:407] Connection header received
0808 02:29:34.236595 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.236599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.236678 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.236788 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:34.238262 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.239050 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.240339 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.240767 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.240909 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.243371 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:34.243376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:34.243379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:34.243410 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:34.246163 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.246906 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.246913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.246916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.247066 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.247464 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.247469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.247472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.247818 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:34.248081 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.248444 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.248697 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 02:29:34.252923 8361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:29:34.260272 8361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:29:34.295167 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.272682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.19.231.3:42775)
0808 02:29:34.285137 (+ 12455us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.285151 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.285219 (+ 68us) server_negotiation.cc:407] Connection header received
0808 02:29:34.285351 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.285362 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.285698 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.287489 (+ 1791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.287810 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.289523 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.292749 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.293948 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.294106 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.294366 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.294653 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.294828 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12216,"thread_start_us":12040,"threads_started":1}
I20260808 02:29:34.295215 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.272413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:42775 (local address 127.0.0.1:52162)
0808 02:29:34.284060 (+ 11647us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.284111 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.284307 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.286686 (+ 2379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.286700 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.286734 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.287248 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.287264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.289639 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.289662 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.292518 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.292544 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.294161 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.294167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.294364 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.294422 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.294571 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.294783 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11399,"mutex_wait_us":32,"thread_start_us":14080,"threads_started":1}
I20260808 02:29:34.296826 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.275021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41573 (local address 127.0.0.1:40938)
0808 02:29:34.287818 (+ 12797us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.287839 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.287960 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.288752 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.288758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.288807 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.289287 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.289297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.291577 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.291583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.294027 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.294044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.296039 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.296045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.296203 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.296248 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.296335 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.296463 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12593,"mutex_wait_us":1,"thread_start_us":12414,"threads_started":1}
I20260808 02:29:34.296998 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.275668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.19.231.2:41573)
0808 02:29:34.287270 (+ 11602us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.287278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.288096 (+ 818us) server_negotiation.cc:407] Connection header received
0808 02:29:34.288197 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.288205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.288499 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.288657 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.289598 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.291319 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.294246 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.295823 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.295991 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.296183 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.296397 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.296604 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11420,"thread_start_us":11306,"threads_started":1}
I20260808 02:29:34.297549 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.272721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:41055 (local address 127.0.0.1:47258)
0808 02:29:34.286964 (+ 14243us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.286989 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.287287 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.288366 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.288370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.288395 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.288958 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.288977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.290578 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.290586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.293018 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.293032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.295675 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.295683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.296922 (+ 1239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.296962 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.297090 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.297249 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13976,"thread_start_us":10956,"threads_started":1}
I20260808 02:29:34.298290 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.273183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.19.231.1:41055)
0808 02:29:34.286514 (+ 13331us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.286523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.287212 (+ 689us) server_negotiation.cc:407] Connection header received
0808 02:29:34.287587 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.287598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.287856 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.288083 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.289226 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.290434 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.293209 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.294864 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.295045 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.296227 (+ 1182us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.297897 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.298036 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13116,"thread_start_us":12908,"threads_started":1}
I20260808 02:29:34.300058 8781 tablet_service.cc:1467] Processing CreateTablet for tablet fe7669072f3f4af88d5fdbfe241d3f96 (DEFAULT_TABLE table=test-table [id=3184add4a9554c59a3a7ddc25da20095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:34.302435 8781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7669072f3f4af88d5fdbfe241d3f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:34.302074 8639 tablet_service.cc:1467] Processing CreateTablet for tablet fe7669072f3f4af88d5fdbfe241d3f96 (DEFAULT_TABLE table=test-table [id=3184add4a9554c59a3a7ddc25da20095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:34.302237 8496 tablet_service.cc:1467] Processing CreateTablet for tablet fe7669072f3f4af88d5fdbfe241d3f96 (DEFAULT_TABLE table=test-table [id=3184add4a9554c59a3a7ddc25da20095]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:34.303256 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7669072f3f4af88d5fdbfe241d3f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:34.303689 8496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7669072f3f4af88d5fdbfe241d3f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:34.315642 8873 tablet_bootstrap.cc:492] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Bootstrap starting.
I20260808 02:29:34.318109 8874 tablet_bootstrap.cc:492] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Bootstrap starting.
I20260808 02:29:34.319777 8873 tablet_bootstrap.cc:654] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:34.319820 8875 tablet_bootstrap.cc:492] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Bootstrap starting.
I20260808 02:29:34.321363 8873 log.cc:824] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:34.322340 8874 tablet_bootstrap.cc:654] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:34.323117 8875 tablet_bootstrap.cc:654] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:34.323912 8874 log.cc:824] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:34.324299 8875 log.cc:824] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:34.325062 8873 tablet_bootstrap.cc:492] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: No bootstrap required, opened a new log
I20260808 02:29:34.325371 8873 ts_tablet_manager.cc:1397] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 02:29:34.326818 8874 tablet_bootstrap.cc:492] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: No bootstrap required, opened a new log
I20260808 02:29:34.327095 8874 ts_tablet_manager.cc:1397] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260808 02:29:34.327615 8875 tablet_bootstrap.cc:492] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: No bootstrap required, opened a new log
I20260808 02:29:34.327889 8875 ts_tablet_manager.cc:1397] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 02:29:34.334946 8874 raft_consensus.cc:348] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.335340 8874 raft_consensus.cc:374] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:34.335451 8874 raft_consensus.cc:729] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a85c470aaa497ea3289ad17ee7ea6c, State: Initialized, Role: FOLLOWER
I20260808 02:29:34.336092 8874 consensus_queue.cc:260] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.337177 8873 raft_consensus.cc:348] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.337716 8873 raft_consensus.cc:374] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:34.337867 8873 raft_consensus.cc:729] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522ed0719a184154b3d3b46dc40b010c, State: Initialized, Role: FOLLOWER
I20260808 02:29:34.338708 8873 consensus_queue.cc:260] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.339871 8875 raft_consensus.cc:348] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.340543 8875 raft_consensus.cc:374] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:34.340737 8875 raft_consensus.cc:729] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee7aab7699a4b85ac82b5da3d50cf61, State: Initialized, Role: FOLLOWER
I20260808 02:29:34.341746 8875 consensus_queue.cc:260] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.343189 8850 heartbeater.cc:503] Master 127.19.231.62:40119 was elected leader, sending a full tablet report...
I20260808 02:29:34.344053 8874 ts_tablet_manager.cc:1428] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Time spent starting tablet: real 0.017s user 0.010s sys 0.000s
I20260808 02:29:34.344086 8875 ts_tablet_manager.cc:1428] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260808 02:29:34.345052 8708 heartbeater.cc:503] Master 127.19.231.62:40119 was elected leader, sending a full tablet report...
I20260808 02:29:34.347391 8873 ts_tablet_manager.cc:1428] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Time spent starting tablet: real 0.022s user 0.013s sys 0.005s
I20260808 02:29:34.349152 8881 raft_consensus.cc:482] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:34.349943 8881 raft_consensus.cc:504] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.351833 8880 raft_consensus.cc:482] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:34.352164 8880 raft_consensus.cc:504] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.364053 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.352585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41573 (local address 127.19.231.3:37477)
0808 02:29:34.352981 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.353005 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.353144 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.354500 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.354506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.354535 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.354799 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.354819 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.356794 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.356804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.359750 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.359776 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.362829 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.362839 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.363171 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.363235 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.363375 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.363586 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
W20260808 02:29:34.367350 8709 tablet.cc:2369] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:29:34.371464 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.353761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:37477 (local address 127.19.231.2:41573)
0808 02:29:34.353982 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.353988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.354022 (+ 34us) server_negotiation.cc:407] Connection header received
0808 02:29:34.354093 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.354099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.354181 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.354280 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.355056 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.356477 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.360305 (+ 3828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.362280 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.362508 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.362702 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.370757 (+ 8055us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.370982 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:29:34.372119 8881 leader_election.cc:288] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45a85c470aaa497ea3289ad17ee7ea6c (127.19.231.1:41055), 522ed0719a184154b3d3b46dc40b010c (127.19.231.2:41573)
I20260808 02:29:34.375969 8880 leader_election.cc:288] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ee7aab7699a4b85ac82b5da3d50cf61 (127.19.231.3:42775), 45a85c470aaa497ea3289ad17ee7ea6c (127.19.231.1:41055)
I20260808 02:29:34.376389 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.353119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:42699 (local address 127.19.231.1:41055)
0808 02:29:34.353501 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.353508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.364898 (+ 11390us) server_negotiation.cc:407] Connection header received
0808 02:29:34.365256 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.365266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.365380 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.365470 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.366329 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.367675 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.372569 (+ 4894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.374472 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.374706 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.374854 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.375749 (+ 895us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.375986 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260808 02:29:34.377864 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.352621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:41055 (local address 127.19.231.3:42699)
0808 02:29:34.364782 (+ 12161us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.364808 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.365041 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.365747 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.365756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.365793 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.366094 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.366110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.367943 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.367948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.370504 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.370522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.375106 (+ 4584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.375117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.375466 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.375521 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.377143 (+ 1622us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.377359 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12019,"threads_started":1}
I20260808 02:29:34.379739 8660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522ed0719a184154b3d3b46dc40b010c" is_pre_election: true
I20260808 02:29:34.380345 8660 raft_consensus.cc:2454] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ee7aab7699a4b85ac82b5da3d50cf61 in term 0.
I20260808 02:29:34.381390 8733 leader_election.cc:302] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee7aab7699a4b85ac82b5da3d50cf61, 522ed0719a184154b3d3b46dc40b010c; no voters:
I20260808 02:29:34.382051 8881 raft_consensus.cc:2781] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:29:34.382305 8881 raft_consensus.cc:482] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:29:34.382499 8881 raft_consensus.cc:3037] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:34.384642 8518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" is_pre_election: true
I20260808 02:29:34.385272 8518 raft_consensus.cc:2454] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ee7aab7699a4b85ac82b5da3d50cf61 in term 0.
I20260808 02:29:34.385500 8879 raft_consensus.cc:482] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:34.385718 8879 raft_consensus.cc:504] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.387697 8879 leader_election.cc:288] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ee7aab7699a4b85ac82b5da3d50cf61 (127.19.231.3:42775), 522ed0719a184154b3d3b46dc40b010c (127.19.231.2:41573)
I20260808 02:29:34.388072 8881 raft_consensus.cc:504] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.389029 8881 leader_election.cc:288] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [CANDIDATE]: Term 1 election: Requested vote from peers 45a85c470aaa497ea3289ad17ee7ea6c (127.19.231.1:41055), 522ed0719a184154b3d3b46dc40b010c (127.19.231.2:41573)
I20260808 02:29:34.390040 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.359871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:54089 (local address 127.19.231.1:41055)
0808 02:29:34.377083 (+ 17212us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.377092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.381885 (+ 4793us) server_negotiation.cc:407] Connection header received
0808 02:29:34.382081 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.382089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.382181 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.382282 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.383286 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.384388 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.386998 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.388495 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.388708 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.388797 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.389541 (+ 744us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.389703 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17049,"thread_start_us":29366,"threads_started":1}
I20260808 02:29:34.390594 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.360116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:42913 (local address 127.19.231.3:42775)
0808 02:29:34.360331 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.360340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.379142 (+ 18802us) server_negotiation.cc:407] Connection header received
0808 02:29:34.379984 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.379994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.380114 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.380198 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.382554 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.383976 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.386904 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.388428 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.388597 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.388691 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.390186 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.390325 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:29:34.390870 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.359283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:41055 (local address 127.19.231.2:54089)
0808 02:29:34.381775 (+ 22492us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.381792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.381941 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.382484 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.382492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.382533 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.383001 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.383017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.384605 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.384610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.386815 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.386830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.389057 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.389062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.389315 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.389354 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.390218 (+ 864us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.390358 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22372,"mutex_wait_us":30,"thread_start_us":19149,"threads_started":1}
I20260808 02:29:34.391809 8518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a85c470aaa497ea3289ad17ee7ea6c"
I20260808 02:29:34.392047 8518 raft_consensus.cc:3037] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:34.392326 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.354502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:42775 (local address 127.19.231.2:42913)
0808 02:29:34.378983 (+ 24481us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.379028 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.379752 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.380456 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.380470 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.380509 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.381143 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.381161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.384184 (+ 3023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.384190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.386674 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.386696 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.389019 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.389028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.389928 (+ 900us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.389990 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.391585 (+ 1595us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.391766 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24271,"mutex_wait_us":644,"thread_start_us":26969,"threads_started":1}
I20260808 02:29:34.393236 8801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "522ed0719a184154b3d3b46dc40b010c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" is_pre_election: true
I20260808 02:29:34.393463 8660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522ed0719a184154b3d3b46dc40b010c"
I20260808 02:29:34.393687 8660 raft_consensus.cc:3037] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:34.393882 8801 raft_consensus.cc:2379] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 522ed0719a184154b3d3b46dc40b010c in current term 1: Already voted for candidate 2ee7aab7699a4b85ac82b5da3d50cf61 in this term.
I20260808 02:29:34.395783 8518 raft_consensus.cc:2454] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ee7aab7699a4b85ac82b5da3d50cf61 in term 1.
I20260808 02:29:34.396582 8734 leader_election.cc:302] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee7aab7699a4b85ac82b5da3d50cf61, 45a85c470aaa497ea3289ad17ee7ea6c; no voters:
I20260808 02:29:34.396874 8881 raft_consensus.cc:2781] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:34.397394 8660 raft_consensus.cc:2454] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ee7aab7699a4b85ac82b5da3d50cf61 in term 1.
I20260808 02:29:34.397435 8881 raft_consensus.cc:686] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 1 LEADER]: Becoming Leader. State: Replica: 2ee7aab7699a4b85ac82b5da3d50cf61, State: Running, Role: LEADER
I20260808 02:29:34.397964 8881 consensus_queue.cc:237] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } }
I20260808 02:29:34.398792 8518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "522ed0719a184154b3d3b46dc40b010c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" is_pre_election: true
I20260808 02:29:34.399106 8518 raft_consensus.cc:2379] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 522ed0719a184154b3d3b46dc40b010c in current term 1: Already voted for candidate 2ee7aab7699a4b85ac82b5da3d50cf61 in this term.
I20260808 02:29:34.401746 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.388752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:39829 (local address 127.19.231.3:42775)
0808 02:29:34.390866 (+ 2114us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.390873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.391525 (+ 652us) server_negotiation.cc:407] Connection header received
0808 02:29:34.391851 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.391856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.391937 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.392004 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.393020 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.394191 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.398775 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.400191 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.400356 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.400465 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.401267 (+ 802us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.401398 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1992,"thread_start_us":2067,"threads_started":1}
I20260808 02:29:34.402491 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.388365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:42775 (local address 127.19.231.1:39829)
0808 02:29:34.391427 (+ 3062us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.391445 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.391637 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.392235 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.392239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.392334 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.392822 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.392839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.394373 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.394378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.396726 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.396741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.400834 (+ 4093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.400841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.401036 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.401082 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.401923 (+ 841us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.402116 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2925,"thread_start_us":2782,"threads_started":1}
I20260808 02:29:34.402491 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.389810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:41463 (local address 127.19.231.2:41573)
0808 02:29:34.390043 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.390052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.392100 (+ 2048us) server_negotiation.cc:407] Connection header received
0808 02:29:34.392427 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.392434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.392550 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.392618 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:34.395910 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.397167 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.399813 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.401182 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.401325 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.401399 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.402091 (+ 692us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.402223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:29:34.402920 8592 leader_election.cc:302] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 522ed0719a184154b3d3b46dc40b010c; no voters: 2ee7aab7699a4b85ac82b5da3d50cf61, 45a85c470aaa497ea3289ad17ee7ea6c
I20260808 02:29:34.403254 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.391319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:41573 (local address 127.19.231.1:41463)
0808 02:29:34.391987 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:34.392011 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:34.392208 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:34.395131 (+ 2923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:34.395137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:34.395165 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:34.395694 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.395708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.397404 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.397411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.399609 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:34.399627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.401680 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.401687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:34.401886 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.401923 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 02:29:34.402711 (+ 788us) client_negotiation.cc:240] Negotiation successful
0808 02:29:34.402885 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":6826,"threads_started":1}
I20260808 02:29:34.403306 8801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" is_pre_election: true
I20260808 02:29:34.404181 8660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7669072f3f4af88d5fdbfe241d3f96" candidate_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522ed0719a184154b3d3b46dc40b010c" is_pre_election: true
I20260808 02:29:34.403676 8359 catalog_manager.cc:5495] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ee7aab7699a4b85ac82b5da3d50cf61 (127.19.231.3). New cstate: current_term: 1 leader_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee7aab7699a4b85ac82b5da3d50cf61" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 42775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 } health_report { overall_health: UNKNOWN } } }
I20260808 02:29:34.404476 8660 raft_consensus.cc:2379] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45a85c470aaa497ea3289ad17ee7ea6c in current term 1: Already voted for candidate 2ee7aab7699a4b85ac82b5da3d50cf61 in this term.
I20260808 02:29:34.405040 8880 raft_consensus.cc:2726] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 02:29:34.405135 8444 leader_election.cc:302] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45a85c470aaa497ea3289ad17ee7ea6c; no voters: 2ee7aab7699a4b85ac82b5da3d50cf61, 522ed0719a184154b3d3b46dc40b010c
I20260808 02:29:34.405589 8879 raft_consensus.cc:2726] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 02:29:34.421026 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.414998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.19.231.3:42775)
0808 02:29:34.415196 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.415203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.415354 (+ 151us) server_negotiation.cc:407] Connection header received
0808 02:29:34.415674 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.415680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.415794 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.415975 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:29:34.416398 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.417144 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.418695 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.419133 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.419279 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.419828 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:29:34.420368 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:29:34.420556 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.420621 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.420707 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:29:34.429234 8518 raft_consensus.cc:1261] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 1 FOLLOWER]: Refusing update from remote peer 2ee7aab7699a4b85ac82b5da3d50cf61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:34.429240 8660 raft_consensus.cc:1261] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 1 FOLLOWER]: Refusing update from remote peer 2ee7aab7699a4b85ac82b5da3d50cf61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:34.430145 8889 consensus_queue.cc:1035] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a85c470aaa497ea3289ad17ee7ea6c" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 41055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:34.430532 8881 consensus_queue.cc:1035] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522ed0719a184154b3d3b46dc40b010c" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:34.455945 8361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:34.456264 8361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:34.460125 8361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b62d70387505411584f963dc5408e80d: Sending DeleteTablet for 3 replicas of tablet fe7669072f3f4af88d5fdbfe241d3f96
I20260808 02:29:34.461326 8781 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7669072f3f4af88d5fdbfe241d3f96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:34 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:52162
I20260808 02:29:34.461345 8496 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7669072f3f4af88d5fdbfe241d3f96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:34 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:47258
I20260808 02:29:34.461740 8639 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7669072f3f4af88d5fdbfe241d3f96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:34 UTC) from {username='kudu', principal='kudu/127.19.231.62@KRBTEST.COM'} at 127.0.0.1:40938
I20260808 02:29:34.463757 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 8424
I20260808 02:29:34.476683 8906 tablet_replica.cc:330] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: stopping tablet replica
I20260808 02:29:34.477385 8907 tablet_replica.cc:330] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: stopping tablet replica
I20260808 02:29:34.478207 8907 raft_consensus.cc:2229] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:34.478926 8906 raft_consensus.cc:2229] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:34.479703 8906 raft_consensus.cc:2258] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:34.480296 8907 raft_consensus.cc:2258] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:34.480614 8905 tablet_replica.cc:330] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: stopping tablet replica
I20260808 02:29:34.481114 8905 raft_consensus.cc:2229] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:29:34.481894 8905 raft_consensus.cc:2258] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:34.483264 8907 ts_tablet_manager.cc:1905] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:34.484118 8905 ts_tablet_manager.cc:1905] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:34.484368 8906 ts_tablet_manager.cc:1905] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:34.490749 8907 ts_tablet_manager.cc:1918] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:34.490993 8907 log.cc:1196] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/fe7669072f3f4af88d5fdbfe241d3f96
I20260808 02:29:34.491693 8907 ts_tablet_manager.cc:1939] T fe7669072f3f4af88d5fdbfe241d3f96 P 522ed0719a184154b3d3b46dc40b010c: Deleting consensus metadata
I20260808 02:29:34.492012 8905 ts_tablet_manager.cc:1918] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:34.492285 8905 log.cc:1196] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/fe7669072f3f4af88d5fdbfe241d3f96
I20260808 02:29:34.492942 8905 ts_tablet_manager.cc:1939] T fe7669072f3f4af88d5fdbfe241d3f96 P 2ee7aab7699a4b85ac82b5da3d50cf61: Deleting consensus metadata
I20260808 02:29:34.494019 8344 catalog_manager.cc:4841] TS 522ed0719a184154b3d3b46dc40b010c (127.19.231.2:41573): tablet fe7669072f3f4af88d5fdbfe241d3f96 (table test-table [id=3184add4a9554c59a3a7ddc25da20095]) successfully deleted
I20260808 02:29:34.494879 8346 catalog_manager.cc:4841] TS 2ee7aab7699a4b85ac82b5da3d50cf61 (127.19.231.3:42775): tablet fe7669072f3f4af88d5fdbfe241d3f96 (table test-table [id=3184add4a9554c59a3a7ddc25da20095]) successfully deleted
I20260808 02:29:34.496356 8906 ts_tablet_manager.cc:1918] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 08 02:29:34 dist-test-slave-pgkr krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.1@KRBTEST.COM
I20260808 02:29:34.496537 8906 log.cc:1196] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/fe7669072f3f4af88d5fdbfe241d3f96
I20260808 02:29:34.497100 8906 ts_tablet_manager.cc:1939] T fe7669072f3f4af88d5fdbfe241d3f96 P 45a85c470aaa497ea3289ad17ee7ea6c: Deleting consensus metadata
I20260808 02:29:34.498941 8345 catalog_manager.cc:4841] TS 45a85c470aaa497ea3289ad17ee7ea6c (127.19.231.1:41055): tablet fe7669072f3f4af88d5fdbfe241d3f96 (table test-table [id=3184add4a9554c59a3a7ddc25da20095]) successfully deleted
I20260808 02:29:34.504392 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.465406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.19.231.1:41055)
0808 02:29:34.465641 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.465650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.477595 (+ 11945us) server_negotiation.cc:407] Connection header received
0808 02:29:34.477896 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.477903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.478015 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.478111 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:34.480710 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.481786 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.487208 (+ 5422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.487900 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.488016 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.497584 (+ 9568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:34.497599 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:34.497613 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:34.497648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:34.501415 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.502150 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.502159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.502163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.502328 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.502762 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.502767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.502771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.503236 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:34.503606 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.503780 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.504024 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 02:29:34.505115 8559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:34.599404 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 8424
I20260808 02:29:34.624866 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 8573
Aug 08 02:29:34 dist-test-slave-pgkr krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.2@KRBTEST.COM
I20260808 02:29:34.641855 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.625668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.19.231.2:41573)
0808 02:29:34.625869 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.625876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.626028 (+ 152us) server_negotiation.cc:407] Connection header received
0808 02:29:34.626246 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.626251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.626342 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.626459 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:34.627868 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.628667 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.629964 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.630390 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.630530 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.635337 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:34.635349 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:34.635364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:34.635402 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:34.638957 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.639660 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.639675 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.639678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.639833 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.640234 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.640239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.640244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.640605 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:34.640957 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.641229 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.641458 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 02:29:34.642599 8702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:34.738277 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 8573
I20260808 02:29:34.769078 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 8714
Aug 08 02:29:34 dist-test-slave-pgkr krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.3@KRBTEST.COM
I20260808 02:29:34.788303 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:34.770075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.19.231.3:42775)
0808 02:29:34.770255 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:34.770259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:34.770277 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:29:34.770445 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:34.770449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:34.770510 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:34.770601 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:34.772702 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.773476 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.775374 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:34.775900 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:34.775994 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:34.781766 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:34.781781 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:34.781796 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:34.781828 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:34.785068 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.785807 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.785814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.785817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.785964 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:34.786389 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:34.786393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:34.786398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:34.786929 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:34.787296 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:34.787581 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 02:29:34.787953 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:29:34.789126 8844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:34.917338 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 8714
I20260808 02:29:34.942092 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 8326
I20260808 02:29:34.943365 8392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:35.058800 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 8326
2026-08-08T02:29:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4715 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T02:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:35Z Disabled control of system clock
I20260808 02:29:35.161214 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40789
--webserver_interface=127.19.231.62
--webserver_port=0
--builtin_ntp_servers=127.19.231.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:29:35.520292 8916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:35.520620 8916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:35.520704 8916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:35.528075 8916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:35.528190 8916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:35.528230 8916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:35.528291 8916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:35.528340 8916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:35.537873 8916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:40789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:35.539294 8916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:35.541108 8916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:35.553562 8924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:35.553642 8926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:35.553597 8923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:35.554899 8916 server_base.cc:1034] running on GCE node
I20260808 02:29:35.555787 8916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:35.557216 8916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:35.558454 8916 hybrid_clock.cc:648] HybridClock initialized: now 1786156175558399 us; error 51 us; skew 500 ppm
I20260808 02:29:35.558913 8916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:35.561383 8916 webserver.cc:460] Webserver started at http://127.19.231.62:34169/ using document root <none> and password file <none>
I20260808 02:29:35.562103 8916 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:35.562238 8916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:35.562582 8916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:35.565492 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f"
format_stamp: "Formatted at 2026-08-08 02:29:35 on dist-test-slave-pgkr"
server_key: "80152b60e97482cb27de985359e40848"
server_key_iv: "7bd3d3cfdd0581085b3cdd70ccab07ae"
server_key_version: "encryptionkey@0"
I20260808 02:29:35.566231 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f"
format_stamp: "Formatted at 2026-08-08 02:29:35 on dist-test-slave-pgkr"
server_key: "80152b60e97482cb27de985359e40848"
server_key_iv: "7bd3d3cfdd0581085b3cdd70ccab07ae"
server_key_version: "encryptionkey@0"
I20260808 02:29:35.571553 8916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 02:29:35.575119 8933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:35.576483 8916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 02:29:35.576792 8916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f"
format_stamp: "Formatted at 2026-08-08 02:29:35 on dist-test-slave-pgkr"
server_key: "80152b60e97482cb27de985359e40848"
server_key_iv: "7bd3d3cfdd0581085b3cdd70ccab07ae"
server_key_version: "encryptionkey@0"
I20260808 02:29:35.577011 8916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:35.606930 8916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:35.607847 8916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:35.608160 8916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:35.628433 8916 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:40789
I20260808 02:29:35.628465 8988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:40789 every 8 connection(s)
I20260808 02:29:35.629886 8916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:35.632483 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 8916
I20260808 02:29:35.632700 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:35.633160 20380 external_mini_cluster.cc:1428] Setting key aa3f014ac35ea8e10df4b27973ce2262
I20260808 02:29:35.634409 8989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:35.648890 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:35.635228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.19.231.62:40789)
0808 02:29:35.636378 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:35.636395 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:35.636454 (+ 59us) server_negotiation.cc:407] Connection header received
0808 02:29:35.638234 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:35.638262 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:35.638951 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:35.639399 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:35.639971 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:35.642071 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:35.643489 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:35.644518 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:35.644719 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:35.645861 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:35.645871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:35.645889 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:35.646715 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:35.646807 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:35.646835 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:35.646966 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:35.647182 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 02:29:35.647575 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":376,"threads_started":1}
I20260808 02:29:35.650003 8989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f: Bootstrap starting.
I20260808 02:29:35.653609 8989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:35.654812 8989 log.cc:824] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:35.657595 8989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f: No bootstrap required, opened a new log
I20260808 02:29:35.666103 8989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40789 } }
I20260808 02:29:35.666527 8989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:35.666677 8989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e05eeeec1d94ed0839c0a4b44ad0b2f, State: Initialized, Role: FOLLOWER
I20260808 02:29:35.667364 8989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40789 } }
I20260808 02:29:35.667623 8989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:35.667780 8989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:35.667979 8989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:35.670269 8989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40789 } }
I20260808 02:29:35.670888 8989 leader_election.cc:302] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e05eeeec1d94ed0839c0a4b44ad0b2f; no voters:
I20260808 02:29:35.671629 8989 leader_election.cc:288] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:35.671737 8994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:35.672497 8994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [term 1 LEADER]: Becoming Leader. State: Replica: 3e05eeeec1d94ed0839c0a4b44ad0b2f, State: Running, Role: LEADER
I20260808 02:29:35.673115 8994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40789 } }
I20260808 02:29:35.673897 8989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:35.676360 8996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e05eeeec1d94ed0839c0a4b44ad0b2f. Latest consensus state: current_term: 1 leader_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40789 } } }
I20260808 02:29:35.676604 8995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e05eeeec1d94ed0839c0a4b44ad0b2f" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40789 } } }
I20260808 02:29:35.676911 8995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:35.676908 8996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:35.679224 9004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:35.685307 9004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:35.694985 9004 catalog_manager.cc:1347] Generated new cluster ID: 93d9a917a25f42a580ec309ae2e2ed91
I20260808 02:29:35.695122 9004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:35.703377 9004 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:35.704651 9004 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:35.723640 9004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f: Generated new TSK 0
I20260808 02:29:35.724758 9004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:35.736572 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.1:0
--local_ip_for_outbound_sockets=127.19.231.1
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40789
--builtin_ntp_servers=127.19.231.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 02:29:36.097380 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:36.097675 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:36.097770 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:36.105126 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:36.105235 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:36.105346 9016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.1
I20260808 02:29:36.114720 9016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.231.1
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.1
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:36.115984 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:36.117725 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:36.130980 9026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:36.131013 9023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:36.130889 9024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:36.131978 9016 server_base.cc:1034] running on GCE node
I20260808 02:29:36.132543 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:36.133438 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:36.134692 9016 hybrid_clock.cc:648] HybridClock initialized: now 1786156176134646 us; error 37 us; skew 500 ppm
I20260808 02:29:36.135037 9016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:36.137337 9016 webserver.cc:460] Webserver started at http://127.19.231.1:36057/ using document root <none> and password file <none>
I20260808 02:29:36.138051 9016 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:36.138177 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:36.138530 9016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:36.141464 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data/instance:
uuid: "d0100ddf0fb945098e11f6c707e6644b"
format_stamp: "Formatted at 2026-08-08 02:29:36 on dist-test-slave-pgkr"
server_key: "27e639b28b66426efa49bc4b2adf0a60"
server_key_iv: "1c226b90da787b7ce5516ddb93153e2f"
server_key_version: "encryptionkey@0"
I20260808 02:29:36.142264 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance:
uuid: "d0100ddf0fb945098e11f6c707e6644b"
format_stamp: "Formatted at 2026-08-08 02:29:36 on dist-test-slave-pgkr"
server_key: "27e639b28b66426efa49bc4b2adf0a60"
server_key_iv: "1c226b90da787b7ce5516ddb93153e2f"
server_key_version: "encryptionkey@0"
I20260808 02:29:36.147428 9016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 02:29:36.151075 9033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:36.152433 9016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 02:29:36.152613 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal
uuid: "d0100ddf0fb945098e11f6c707e6644b"
format_stamp: "Formatted at 2026-08-08 02:29:36 on dist-test-slave-pgkr"
server_key: "27e639b28b66426efa49bc4b2adf0a60"
server_key_iv: "1c226b90da787b7ce5516ddb93153e2f"
server_key_version: "encryptionkey@0"
I20260808 02:29:36.152829 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:36.176802 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:36.177709 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:36.178035 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:36.179260 9016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:36.181048 9016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:36.181152 9016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:36.181236 9016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:36.181295 9016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:36.217523 9016 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.1:37709
I20260808 02:29:36.217638 9161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.1:37709 every 8 connection(s)
I20260808 02:29:36.219003 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/data/info.pb
I20260808 02:29:36.221423 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 9016
I20260808 02:29:36.221613 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal/instance
I20260808 02:29:36.222205 20380 external_mini_cluster.cc:1428] Setting key 0dcc1398a14c6844d063966100f5204a
I20260808 02:29:36.225886 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.2:0
--local_ip_for_outbound_sockets=127.19.231.2
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40789
--builtin_ntp_servers=127.19.231.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:29:36.235167 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:36.221889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:38869 (local address 127.19.231.62:40789)
0808 02:29:36.222602 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:36.222630 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:36.224455 (+ 1825us) server_negotiation.cc:407] Connection header received
0808 02:29:36.227218 (+ 2763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:36.227223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:36.227290 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:36.227388 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:36.228901 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.229725 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.232208 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.232626 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.232738 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:36.234351 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:36.234355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:36.234358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:36.234376 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:36.234417 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:36.234429 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:36.234501 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:36.234796 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 02:29:36.234879 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260808 02:29:36.236697 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:36.222534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40789 (local address 127.19.231.1:38869)
0808 02:29:36.224278 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:36.224332 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:36.226912 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:36.227680 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:36.227695 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:36.227831 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:36.228679 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:36.228708 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.229906 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.229914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:36.232047 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:36.232071 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.232863 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.232872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:36.233218 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:36.233991 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:36.233996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:36.234049 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:36.234158 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:36.234529 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:36.234543 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:36.234560 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 02:29:36.234776 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 02:29:36.235215 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"thread_start_us":894,"threads_started":1}
I20260808 02:29:36.238268 9162 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40789
I20260808 02:29:36.238655 9162 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:36.239589 9162 heartbeater.cc:511] Master 127.19.231.62:40789 requested a full tablet report, sending...
I20260808 02:29:36.242161 8951 ts_manager.cc:194] Registered new tserver with Master: d0100ddf0fb945098e11f6c707e6644b (127.19.231.1:37709)
I20260808 02:29:36.244431 8951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.1:38869
W20260808 02:29:36.586422 9167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:36.586854 9167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:36.587028 9167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:36.595355 9167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:36.595527 9167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:36.595733 9167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.2
I20260808 02:29:36.605345 9167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.231.2
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.2
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:36.606850 9167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:36.608832 9167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:36.622764 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:36.622810 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:36.622836 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:36.624061 9167 server_base.cc:1034] running on GCE node
I20260808 02:29:36.624760 9167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:36.625834 9167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:36.627092 9167 hybrid_clock.cc:648] HybridClock initialized: now 1786156176627055 us; error 50 us; skew 500 ppm
I20260808 02:29:36.627435 9167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:36.630079 9167 webserver.cc:460] Webserver started at http://127.19.231.2:36951/ using document root <none> and password file <none>
I20260808 02:29:36.630841 9167 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:36.630975 9167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:36.631309 9167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:36.634218 9167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data/instance:
uuid: "20bdd98fcfb9411eb085ba74e191e849"
format_stamp: "Formatted at 2026-08-08 02:29:36 on dist-test-slave-pgkr"
server_key: "28e82ad1d3bb091f95944bf3d1a6c7a8"
server_key_iv: "50e9a11e3224bdc263490e889f46ec34"
server_key_version: "encryptionkey@0"
I20260808 02:29:36.635090 9167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance:
uuid: "20bdd98fcfb9411eb085ba74e191e849"
format_stamp: "Formatted at 2026-08-08 02:29:36 on dist-test-slave-pgkr"
server_key: "28e82ad1d3bb091f95944bf3d1a6c7a8"
server_key_iv: "50e9a11e3224bdc263490e889f46ec34"
server_key_version: "encryptionkey@0"
I20260808 02:29:36.640939 9167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 02:29:36.644562 9182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:36.646098 9167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:29:36.646245 9167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal
uuid: "20bdd98fcfb9411eb085ba74e191e849"
format_stamp: "Formatted at 2026-08-08 02:29:36 on dist-test-slave-pgkr"
server_key: "28e82ad1d3bb091f95944bf3d1a6c7a8"
server_key_iv: "50e9a11e3224bdc263490e889f46ec34"
server_key_version: "encryptionkey@0"
I20260808 02:29:36.646433 9167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:36.662328 9167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:36.663168 9167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:36.663437 9167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:36.664382 9167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:36.665836 9167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:36.665925 9167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:36.666015 9167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:36.666071 9167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:36.699812 9167 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.2:34791
I20260808 02:29:36.699862 9301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.2:34791 every 8 connection(s)
I20260808 02:29:36.701174 9167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/data/info.pb
I20260808 02:29:36.706277 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 9167
I20260808 02:29:36.706459 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal/instance
I20260808 02:29:36.706965 20380 external_mini_cluster.cc:1428] Setting key 02c200fbf9912335bfbe61d9fb8ced82
I20260808 02:29:36.710155 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.231.3:0
--local_ip_for_outbound_sockets=127.19.231.3
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40789
--builtin_ntp_servers=127.19.231.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 02:29:36.714931 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:36.703768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.2:50605 (local address 127.19.231.62:40789)
0808 02:29:36.703980 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:36.703986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:36.705406 (+ 1420us) server_negotiation.cc:407] Connection header received
0808 02:29:36.707457 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:36.707463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:36.707551 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:36.707639 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:36.708857 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.709905 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.711701 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.712428 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.712573 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:36.713964 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:36.713972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:36.713979 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:36.714003 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:36.714046 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:36.714066 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:36.714159 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:36.714409 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 02:29:36.714515 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:29:36.715905 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:36.704351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40789 (local address 127.19.231.2:50605)
0808 02:29:36.705254 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:36.705290 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:36.707184 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:36.708012 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:36.708020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:36.708113 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:36.708674 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:36.708691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.710110 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.710114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:36.711541 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:36.711557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:36.712655 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:36.712660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:36.712869 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:36.713634 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:36.713637 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:36.713678 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:36.713757 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:36.714178 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:36.714188 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:36.714195 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 02:29:36.714372 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 02:29:36.714760 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":282,"threads_started":1}
I20260808 02:29:36.717373 9302 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40789
I20260808 02:29:36.717706 9302 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:36.718684 9302 heartbeater.cc:511] Master 127.19.231.62:40789 requested a full tablet report, sending...
I20260808 02:29:36.720300 8951 ts_manager.cc:194] Registered new tserver with Master: 20bdd98fcfb9411eb085ba74e191e849 (127.19.231.2:34791)
I20260808 02:29:36.721149 8951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.2:50605
W20260808 02:29:37.059172 9305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:37.059507 9305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:37.059684 9305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:37.067024 9305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:37.067158 9305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:37.067338 9305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.231.3
I20260808 02:29:37.076661 9305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.231.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.231.3
--webserver_port=0
--tserver_master_addrs=127.19.231.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.231.3
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:37.078027 9305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:37.079965 9305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:37.093328 9310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:37.093374 9311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:37.093379 9313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:37.094322 9305 server_base.cc:1034] running on GCE node
I20260808 02:29:37.095093 9305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:37.096055 9305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:37.097298 9305 hybrid_clock.cc:648] HybridClock initialized: now 1786156177097253 us; error 61 us; skew 500 ppm
I20260808 02:29:37.097669 9305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 02:29:37.100087 9305 webserver.cc:460] Webserver started at http://127.19.231.3:36419/ using document root <none> and password file <none>
I20260808 02:29:37.100826 9305 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:37.100947 9305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:37.101293 9305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:37.104260 9305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data/instance:
uuid: "0dca3cdadc66415f82ea08698b6e0f91"
format_stamp: "Formatted at 2026-08-08 02:29:37 on dist-test-slave-pgkr"
server_key: "1d1e490ff555b9aebba78b88ea2f2973"
server_key_iv: "0f2824b1aa5a76325d201c7e0e10c6b8"
server_key_version: "encryptionkey@0"
I20260808 02:29:37.105058 9305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance:
uuid: "0dca3cdadc66415f82ea08698b6e0f91"
format_stamp: "Formatted at 2026-08-08 02:29:37 on dist-test-slave-pgkr"
server_key: "1d1e490ff555b9aebba78b88ea2f2973"
server_key_iv: "0f2824b1aa5a76325d201c7e0e10c6b8"
server_key_version: "encryptionkey@0"
I20260808 02:29:37.110368 9305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:29:37.114082 9319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:37.115477 9305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:29:37.115664 9305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal
uuid: "0dca3cdadc66415f82ea08698b6e0f91"
format_stamp: "Formatted at 2026-08-08 02:29:37 on dist-test-slave-pgkr"
server_key: "1d1e490ff555b9aebba78b88ea2f2973"
server_key_iv: "0f2824b1aa5a76325d201c7e0e10c6b8"
server_key_version: "encryptionkey@0"
I20260808 02:29:37.115900 9305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:37.130002 9305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:37.130873 9305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:37.131175 9305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:37.132126 9305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 02:29:37.133625 9305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 02:29:37.133751 9305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:37.133888 9305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 02:29:37.133980 9305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:37.168871 9305 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.3:34599
I20260808 02:29:37.168964 9440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.3:34599 every 8 connection(s)
I20260808 02:29:37.170254 9305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/data/info.pb
I20260808 02:29:37.175719 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 9305
I20260808 02:29:37.175928 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal/instance
I20260808 02:29:37.176384 20380 external_mini_cluster.cc:1428] Setting key 37346325df7f9384918da1a2c0050359
I20260808 02:29:37.183960 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.173170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.3:38319 (local address 127.19.231.62:40789)
0808 02:29:37.173363 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.173368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.174968 (+ 1600us) server_negotiation.cc:407] Connection header received
0808 02:29:37.176965 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.176971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.177070 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.177185 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:37.178324 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.179327 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.180993 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.181498 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.181635 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.183103 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:37.183109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:37.183114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:37.183142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:37.183176 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:37.183188 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:37.183294 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.183520 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.183619 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:29:37.184973 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.173781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:40789 (local address 127.19.231.3:38319)
0808 02:29:37.174803 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.174857 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:37.176676 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:37.177438 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:37.177446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:37.177535 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:29:37.178144 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.178175 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.179490 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.179495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.180843 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.180855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.181735 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.181742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.181972 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.182776 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 02:29:37.182779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:29:37.182819 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 02:29:37.182898 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:29:37.183295 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:29:37.183310 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:29:37.183319 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 02:29:37.183508 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 02:29:37.183920 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":265,"threads_started":1}
I20260808 02:29:37.186300 9441 heartbeater.cc:347] Connected to a master server at 127.19.231.62:40789
I20260808 02:29:37.186578 9441 heartbeater.cc:464] Registering TS with master...
I20260808 02:29:37.187333 9441 heartbeater.cc:511] Master 127.19.231.62:40789 requested a full tablet report, sending...
I20260808 02:29:37.188760 8951 ts_manager.cc:194] Registered new tserver with Master: 0dca3cdadc66415f82ea08698b6e0f91 (127.19.231.3:34599)
I20260808 02:29:37.189626 8951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.231.3:38319
I20260808 02:29:37.191551 20380 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 02:29:37.201717 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.195739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.19.231.62:40789)
0808 02:29:37.195961 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.195967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.196270 (+ 303us) server_negotiation.cc:407] Connection header received
0808 02:29:37.196463 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.196469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.196555 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.196660 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:37.197176 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.197881 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.199349 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.199775 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.199911 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.200997 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:37.201003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:37.201008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:37.201035 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:37.201064 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:37.201076 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:37.201172 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.201329 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.201418 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:29:37.205654 8951 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 02:29:37.212632 8951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 02:29:37.244870 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.226840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.19.231.3:34599)
0808 02:29:37.227609 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.227621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.231637 (+ 4016us) server_negotiation.cc:407] Connection header received
0808 02:29:37.231859 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.231871 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.232207 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.232439 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:37.234955 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.237091 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.241967 (+ 4876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.243554 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.243709 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.243994 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.244277 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.244450 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":414,"threads_started":1}
I20260808 02:29:37.245697 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.226146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:34599 (local address 127.0.0.1:39872)
0808 02:29:37.231517 (+ 5371us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.231531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:37.231669 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:37.234086 (+ 2417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:37.234097 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:37.234139 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:37.234683 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.234700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.238497 (+ 3797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.238505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.241692 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.241713 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.243764 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.243773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.243987 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.244047 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 02:29:37.245160 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0808 02:29:37.245372 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5232,"thread_start_us":3955,"threads_started":1}
I20260808 02:29:37.247881 9162 heartbeater.cc:503] Master 127.19.231.62:40789 was elected leader, sending a full tablet report...
I20260808 02:29:37.251259 9370 tablet_service.cc:1467] Processing CreateTablet for tablet f56bdfd97c844c4790b1ab863e598586 (DEFAULT_TABLE table=test-table [id=0599790aa337435686941a2283aec624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:37.253217 9370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56bdfd97c844c4790b1ab863e598586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:37.253744 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.226100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:34791 (local address 127.0.0.1:43278)
0808 02:29:37.230646 (+ 4546us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.230708 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:37.230916 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:37.243249 (+ 12333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:37.243257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:37.243292 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:37.243792 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.243807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.246188 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.246194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.248548 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.248564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.252754 (+ 4190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.252766 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.252958 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.253006 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 02:29:37.253132 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 02:29:37.253332 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4215,"mutex_wait_us":3,"thread_start_us":5157,"threads_started":1}
I20260808 02:29:37.254184 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.226151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.19.231.2:34791)
0808 02:29:37.241491 (+ 15340us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.241503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.241543 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:29:37.241664 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.241675 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.241990 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.244053 (+ 2063us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:37.244303 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.246017 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.248805 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.250307 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.250499 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.250802 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.253265 (+ 2463us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.253517 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15111,"thread_start_us":16766,"threads_started":1}
I20260808 02:29:37.258522 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.231791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54246)
0808 02:29:37.242653 (+ 10862us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.242676 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:37.242821 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:37.246112 (+ 3291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:37.246117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:37.246148 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:37.251288 (+ 5140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.251299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.253088 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.253096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.255450 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.255463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.257688 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.257695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.257870 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.257915 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 02:29:37.258006 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 02:29:37.258152 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10698,"mutex_wait_us":81,"thread_start_us":10543,"threads_started":1}
I20260808 02:29:37.258692 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.231522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.19.231.1:37709)
0808 02:29:37.244504 (+ 12982us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.244512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.244576 (+ 64us) server_negotiation.cc:407] Connection header received
0808 02:29:37.244734 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.244744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.245530 (+ 786us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.246964 (+ 1434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:37.251512 (+ 4548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.252948 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.255929 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.257481 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.257653 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.257935 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.258063 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.258223 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12766,"thread_start_us":12628,"threads_started":1}
I20260808 02:29:37.258929 9231 tablet_service.cc:1467] Processing CreateTablet for tablet f56bdfd97c844c4790b1ab863e598586 (DEFAULT_TABLE table=test-table [id=0599790aa337435686941a2283aec624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:37.260509 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56bdfd97c844c4790b1ab863e598586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:37.263835 9086 tablet_service.cc:1467] Processing CreateTablet for tablet f56bdfd97c844c4790b1ab863e598586 (DEFAULT_TABLE table=test-table [id=0599790aa337435686941a2283aec624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 02:29:37.265367 9086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56bdfd97c844c4790b1ab863e598586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:37.269160 9459 tablet_bootstrap.cc:492] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Bootstrap starting.
I20260808 02:29:37.273249 9459 tablet_bootstrap.cc:654] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:37.274219 9460 tablet_bootstrap.cc:492] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Bootstrap starting.
I20260808 02:29:37.274690 9459 log.cc:824] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:37.277711 9460 tablet_bootstrap.cc:654] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:37.278940 9460 log.cc:824] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:37.281780 9459 tablet_bootstrap.cc:492] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: No bootstrap required, opened a new log
I20260808 02:29:37.282032 9463 tablet_bootstrap.cc:492] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: Bootstrap starting.
I20260808 02:29:37.282105 9459 ts_tablet_manager.cc:1397] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260808 02:29:37.282886 9460 tablet_bootstrap.cc:492] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: No bootstrap required, opened a new log
I20260808 02:29:37.283170 9460 ts_tablet_manager.cc:1397] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 02:29:37.285584 9463 tablet_bootstrap.cc:654] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:37.286966 9463 log.cc:824] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:37.290755 9463 tablet_bootstrap.cc:492] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: No bootstrap required, opened a new log
I20260808 02:29:37.291136 9463 ts_tablet_manager.cc:1397] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260808 02:29:37.292201 9459 raft_consensus.cc:348] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.292737 9459 raft_consensus.cc:374] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:37.292860 9459 raft_consensus.cc:729] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dca3cdadc66415f82ea08698b6e0f91, State: Initialized, Role: FOLLOWER
I20260808 02:29:37.292740 9460 raft_consensus.cc:348] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.293203 9460 raft_consensus.cc:374] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:37.293325 9460 raft_consensus.cc:729] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20bdd98fcfb9411eb085ba74e191e849, State: Initialized, Role: FOLLOWER
I20260808 02:29:37.293576 9459 consensus_queue.cc:260] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.293922 9460 consensus_queue.cc:260] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.294998 9441 heartbeater.cc:503] Master 127.19.231.62:40789 was elected leader, sending a full tablet report...
I20260808 02:29:37.295500 9459 ts_tablet_manager.cc:1428] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260808 02:29:37.296415 9302 heartbeater.cc:503] Master 127.19.231.62:40789 was elected leader, sending a full tablet report...
I20260808 02:29:37.297009 9460 ts_tablet_manager.cc:1428] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260808 02:29:37.302147 9463 raft_consensus.cc:348] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.302595 9463 raft_consensus.cc:374] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:37.302748 9463 raft_consensus.cc:729] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0100ddf0fb945098e11f6c707e6644b, State: Initialized, Role: FOLLOWER
I20260808 02:29:37.303344 9463 consensus_queue.cc:260] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.304876 9463 ts_tablet_manager.cc:1428] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260808 02:29:37.316663 9467 raft_consensus.cc:482] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:37.316905 9467 raft_consensus.cc:504] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.318477 9467 leader_election.cc:288] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20bdd98fcfb9411eb085ba74e191e849 (127.19.231.2:34791), 0dca3cdadc66415f82ea08698b6e0f91 (127.19.231.3:34599)
I20260808 02:29:37.325685 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.319095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:32869 (local address 127.19.231.3:34599)
0808 02:29:37.319486 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.319492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.320003 (+ 511us) server_negotiation.cc:407] Connection header received
0808 02:29:37.320238 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.320244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.320342 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.320445 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:37.321069 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.321812 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.323744 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.324773 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.324947 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.325013 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.325353 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.325480 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 02:29:37.325924 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.319047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.231.1:39141 (local address 127.19.231.2:34791)
0808 02:29:37.319552 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.319558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.320182 (+ 624us) server_negotiation.cc:407] Connection header received
0808 02:29:37.320421 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.320427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.320513 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.320609 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 02:29:37.321268 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.321949 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.323820 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.324763 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.324917 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.324981 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.325547 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.325669 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371}
I20260808 02:29:37.326027 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.319102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.2:34791 (local address 127.19.231.1:39141)
0808 02:29:37.320084 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.320100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:37.320229 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:37.320640 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:37.320645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:37.320688 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:37.321099 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.321108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.322107 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.322112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.323635 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.323648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.325137 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.325143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.325324 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.325353 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 02:29:37.325513 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 02:29:37.325647 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":48,"thread_start_us":640,"threads_started":1}
I20260808 02:29:37.326438 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.318855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.3:34599 (local address 127.19.231.1:32869)
0808 02:29:37.319864 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.319897 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 02:29:37.320060 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:29:37.320493 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:29:37.320498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:29:37.320527 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 02:29:37.320917 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.320929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.321943 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.321949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.323552 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:29:37.323564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.324957 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.324963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:29:37.325119 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.325183 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 02:29:37.325924 (+ 741us) client_negotiation.cc:240] Negotiation successful
0808 02:29:37.326112 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":4,"spinlock_wait_cycles":896,"thread_start_us":517,"threads_started":1}
I20260808 02:29:37.327090 9254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56bdfd97c844c4790b1ab863e598586" candidate_uuid: "d0100ddf0fb945098e11f6c707e6644b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20bdd98fcfb9411eb085ba74e191e849" is_pre_election: true
I20260808 02:29:37.327435 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56bdfd97c844c4790b1ab863e598586" candidate_uuid: "d0100ddf0fb945098e11f6c707e6644b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dca3cdadc66415f82ea08698b6e0f91" is_pre_election: true
I20260808 02:29:37.327580 9254 raft_consensus.cc:2454] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0100ddf0fb945098e11f6c707e6644b in term 0.
I20260808 02:29:37.327924 9391 raft_consensus.cc:2454] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0100ddf0fb945098e11f6c707e6644b in term 0.
I20260808 02:29:37.328234 9039 leader_election.cc:302] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bdd98fcfb9411eb085ba74e191e849, d0100ddf0fb945098e11f6c707e6644b; no voters:
I20260808 02:29:37.328660 9467 raft_consensus.cc:2781] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 02:29:37.328814 9467 raft_consensus.cc:482] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 02:29:37.328935 9467 raft_consensus.cc:3037] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:37.331215 9467 raft_consensus.cc:504] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.331796 9467 leader_election.cc:288] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [CANDIDATE]: Term 1 election: Requested vote from peers 20bdd98fcfb9411eb085ba74e191e849 (127.19.231.2:34791), 0dca3cdadc66415f82ea08698b6e0f91 (127.19.231.3:34599)
I20260808 02:29:37.332244 9391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56bdfd97c844c4790b1ab863e598586" candidate_uuid: "d0100ddf0fb945098e11f6c707e6644b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dca3cdadc66415f82ea08698b6e0f91"
I20260808 02:29:37.332244 9254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56bdfd97c844c4790b1ab863e598586" candidate_uuid: "d0100ddf0fb945098e11f6c707e6644b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20bdd98fcfb9411eb085ba74e191e849"
I20260808 02:29:37.332441 9391 raft_consensus.cc:3037] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:37.332481 9254 raft_consensus.cc:3037] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:37.334692 9391 raft_consensus.cc:2454] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0100ddf0fb945098e11f6c707e6644b in term 1.
I20260808 02:29:37.335006 9254 raft_consensus.cc:2454] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0100ddf0fb945098e11f6c707e6644b in term 1.
I20260808 02:29:37.335125 9035 leader_election.cc:302] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dca3cdadc66415f82ea08698b6e0f91, d0100ddf0fb945098e11f6c707e6644b; no voters:
I20260808 02:29:37.335489 9467 raft_consensus.cc:2781] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:37.335969 9467 raft_consensus.cc:686] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 1 LEADER]: Becoming Leader. State: Replica: d0100ddf0fb945098e11f6c707e6644b, State: Running, Role: LEADER
I20260808 02:29:37.336500 9467 consensus_queue.cc:237] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } }
I20260808 02:29:37.341176 8951 catalog_manager.cc:5495] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b reported cstate change: term changed from 0 to 1, leader changed from <none> to d0100ddf0fb945098e11f6c707e6644b (127.19.231.1). New cstate: current_term: 1 leader_uuid: "d0100ddf0fb945098e11f6c707e6644b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0100ddf0fb945098e11f6c707e6644b" member_type: VOTER last_known_addr { host: "127.19.231.1" port: 37709 } health_report { overall_health: HEALTHY } } }
I20260808 02:29:37.361779 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.356584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.19.231.1:37709)
0808 02:29:37.356831 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.356835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.356853 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:29:37.357068 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.357073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.357190 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.357320 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:29:37.357642 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.358424 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.359836 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.360270 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.360366 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.360810 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:29:37.361328 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:29:37.361443 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.361511 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.361587 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 02:29:37.370651 9391 raft_consensus.cc:1261] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 1 FOLLOWER]: Refusing update from remote peer d0100ddf0fb945098e11f6c707e6644b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:37.370674 9254 raft_consensus.cc:1261] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 1 FOLLOWER]: Refusing update from remote peer d0100ddf0fb945098e11f6c707e6644b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:29:37.371672 9470 consensus_queue.cc:1035] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bdd98fcfb9411eb085ba74e191e849" member_type: VOTER last_known_addr { host: "127.19.231.2" port: 34791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:37.372051 9467 consensus_queue.cc:1035] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dca3cdadc66415f82ea08698b6e0f91" member_type: VOTER last_known_addr { host: "127.19.231.3" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:29:37.383729 9473 mvcc.cc:205] Tried to move back new op lower bound from 7316095702498775040 to 7316095702374612992. Current Snapshot: MvccSnapshot[applied={T|T < 7316095702498775040}]
I20260808 02:29:37.384594 9475 mvcc.cc:205] Tried to move back new op lower bound from 7316095702498775040 to 7316095702374612992. Current Snapshot: MvccSnapshot[applied={T|T < 7316095702498775040}]
I20260808 02:29:37.399993 8950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:37.400314 8950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 02:29:37.404358 8950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e05eeeec1d94ed0839c0a4b44ad0b2f: Sending DeleteTablet for 3 replicas of tablet f56bdfd97c844c4790b1ab863e598586
I20260808 02:29:37.405651 9231 tablet_service.cc:1514] Processing DeleteTablet for tablet f56bdfd97c844c4790b1ab863e598586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:37 UTC) from {username='slave'} at 127.0.0.1:43278
I20260808 02:29:37.405714 9370 tablet_service.cc:1514] Processing DeleteTablet for tablet f56bdfd97c844c4790b1ab863e598586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:37 UTC) from {username='slave'} at 127.0.0.1:39872
I20260808 02:29:37.405942 9086 tablet_service.cc:1514] Processing DeleteTablet for tablet f56bdfd97c844c4790b1ab863e598586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 02:29:37 UTC) from {username='slave'} at 127.0.0.1:54246
I20260808 02:29:37.407296 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9016
I20260808 02:29:37.415269 9484 tablet_replica.cc:330] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: stopping tablet replica
I20260808 02:29:37.415434 9485 tablet_replica.cc:330] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: stopping tablet replica
I20260808 02:29:37.415786 9484 raft_consensus.cc:2229] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:37.415879 9485 raft_consensus.cc:2229] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 1 LEADER]: Raft consensus shutting down.
I20260808 02:29:37.416314 9484 raft_consensus.cc:2258] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:37.416651 9485 raft_consensus.cc:2258] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:37.418936 9485 ts_tablet_manager.cc:1905] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:37.418928 9484 ts_tablet_manager.cc:1905] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:37.419914 9483 tablet_replica.cc:330] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: stopping tablet replica
I20260808 02:29:37.424981 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.408334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.19.231.1:37709)
0808 02:29:37.408546 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.408555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.414100 (+ 5545us) server_negotiation.cc:407] Connection header received
0808 02:29:37.415082 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.415090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.415223 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.415313 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:37.416596 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.417975 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.420322 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.421118 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.421234 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.422945 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:37.422960 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:37.422977 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:37.423864 (+ 887us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:37.423975 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:37.424005 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:37.424106 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.424394 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.424533 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:29:37.426429 9157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:37.426517 9484 ts_tablet_manager.cc:1918] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:37.420406 9483 raft_consensus.cc:2229] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 02:29:37.426755 9484 log.cc:1196] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-2/wal/wals/f56bdfd97c844c4790b1ab863e598586
I20260808 02:29:37.427165 9483 raft_consensus.cc:2258] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 02:29:37.427491 9484 ts_tablet_manager.cc:1939] T f56bdfd97c844c4790b1ab863e598586 P 0dca3cdadc66415f82ea08698b6e0f91: Deleting consensus metadata
I20260808 02:29:37.429909 9483 ts_tablet_manager.cc:1905] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 02:29:37.430477 8935 catalog_manager.cc:4841] TS 0dca3cdadc66415f82ea08698b6e0f91 (127.19.231.3:34599): tablet f56bdfd97c844c4790b1ab863e598586 (table test-table [id=0599790aa337435686941a2283aec624]) successfully deleted
I20260808 02:29:37.438210 9483 ts_tablet_manager.cc:1918] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:37.438429 9483 log.cc:1196] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-1/wal/wals/f56bdfd97c844c4790b1ab863e598586
I20260808 02:29:37.439128 9483 ts_tablet_manager.cc:1939] T f56bdfd97c844c4790b1ab863e598586 P 20bdd98fcfb9411eb085ba74e191e849: Deleting consensus metadata
I20260808 02:29:37.441351 8937 catalog_manager.cc:4841] TS 20bdd98fcfb9411eb085ba74e191e849 (127.19.231.2:34791): tablet f56bdfd97c844c4790b1ab863e598586 (table test-table [id=0599790aa337435686941a2283aec624]) successfully deleted
I20260808 02:29:37.527009 9485 ts_tablet_manager.cc:1918] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 02:29:37.527343 9485 log.cc:1196] T f56bdfd97c844c4790b1ab863e598586 P d0100ddf0fb945098e11f6c707e6644b: Deleting WAL directory at /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786155839876607-20380-0/minicluster-data/ts-0/wal/wals/f56bdfd97c844c4790b1ab863e598586
I20260808 02:29:37.527920 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 9016
W20260808 02:29:37.551080 8935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.231.1:37709 (error 108)
I20260808 02:29:37.552111 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9167
I20260808 02:29:37.553236 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.552761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54282)
0808 02:29:37.552956 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.553089 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 02:29:37.553622 8935 catalog_manager.cc:4584] TS d0100ddf0fb945098e11f6c707e6644b (127.19.231.1:37709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f56bdfd97c844c4790b1ab863e598586: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
I20260808 02:29:37.559465 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.552914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.19.231.2:34791)
0808 02:29:37.553075 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.553082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.553251 (+ 169us) server_negotiation.cc:407] Connection header received
0808 02:29:37.553492 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.553496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.553597 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.553687 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:37.554185 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.555027 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.556288 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.556689 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.556787 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.557962 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:37.557971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:37.557986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:37.558667 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:37.558731 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:37.558760 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:37.558864 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.559062 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.559165 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 02:29:37.560178 9297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:37.607529 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.606939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54284)
0808 02:29:37.607261 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.607365 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260808 02:29:37.608599 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.608251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54292)
0808 02:29:37.608412 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.608481 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 02:29:37.657732 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 9167
I20260808 02:29:37.669019 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.668525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54300)
0808 02:29:37.668758 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.668845 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:29:37.670147 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.669818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54316)
0808 02:29:37.669974 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.670040 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 02:29:37.679641 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9305
I20260808 02:29:37.686710 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.680501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.19.231.3:34599)
0808 02:29:37.680678 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:37.680682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:37.680701 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:29:37.680925 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:37.680931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:37.681024 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:37.681123 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:37.681676 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.682412 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.683692 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:37.684084 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:37.684188 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:37.685228 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:37.685237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:37.685251 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:37.685909 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:37.685956 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:37.685975 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:37.686069 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:37.686198 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:29:37.686282 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 02:29:37.687369 9436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:37.751091 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.750550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54328)
0808 02:29:37.750814 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.750912 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260808 02:29:37.752384 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:37.751837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.1:37709 (local address 127.0.0.1:54342)
0808 02:29:37.752130 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 02:29:37.752226 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.1:37709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260808 02:29:37.777369 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 9305
I20260808 02:29:37.798818 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 8916
I20260808 02:29:37.800104 8984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:37.915539 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 8916
2026-08-08T02:29:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2859 ms)
[----------] 6 tests from EncryptionPolicyTest (23196 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:38 dist-test-slave-pgkr krb5kdc[9494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:38 dist-test-slave-pgkr krb5kdc[9494](info): set up 2 sockets
Aug 08 02:29:38 dist-test-slave-pgkr krb5kdc[9494](info): commencing operation
krb5kdc: starting...
W20260808 02:29:39.116423 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:39 dist-test-slave-pgkr krb5kdc[9494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:39.470986 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:39.829221 9510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:39.829514 9510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:39.829612 9510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:39.837009 9510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:39.837132 9510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:39.837188 9510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:39.837252 9510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:39.837297 9510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:39.847472 9510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:39.848840 9510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:39.850150 9510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:39.862408 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:39.862373 9518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:39.862457 9517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:39.863616 9510 server_base.cc:1034] running on GCE node
I20260808 02:29:39.864467 9510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:39.865912 9510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:39.867161 9510 hybrid_clock.cc:648] HybridClock initialized: now 1786156179867119 us; error 50 us; skew 500 ppm
I20260808 02:29:39.867610 9510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:39 dist-test-slave-pgkr krb5kdc[9494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:39.877133 9510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:39.878952 9510 webserver.cc:460] Webserver started at http://127.0.0.1:43245/ using document root <none> and password file <none>
I20260808 02:29:39.879717 9510 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:39.879833 9510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:39.880121 9510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:39.883908 9510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "798d0882bb4545d88e2e8889ee87ac1c"
format_stamp: "Formatted at 2026-08-08 02:29:39 on dist-test-slave-pgkr"
server_key: "20ab82f8eeed6527319ecb2e188a546b"
server_key_iv: "cde97dd63c0f6d9ff728bf9f35368d32"
server_key_version: "encryptionkey@0"
I20260808 02:29:39.884785 9510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "798d0882bb4545d88e2e8889ee87ac1c"
format_stamp: "Formatted at 2026-08-08 02:29:39 on dist-test-slave-pgkr"
server_key: "20ab82f8eeed6527319ecb2e188a546b"
server_key_iv: "cde97dd63c0f6d9ff728bf9f35368d32"
server_key_version: "encryptionkey@0"
I20260808 02:29:39.890441 9510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:29:39.894225 9528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:39.895941 9510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:39.896153 9510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "798d0882bb4545d88e2e8889ee87ac1c"
format_stamp: "Formatted at 2026-08-08 02:29:39 on dist-test-slave-pgkr"
server_key: "20ab82f8eeed6527319ecb2e188a546b"
server_key_iv: "cde97dd63c0f6d9ff728bf9f35368d32"
server_key_version: "encryptionkey@0"
I20260808 02:29:39.896391 9510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:39.934156 9510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:39.938773 9510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:39.939045 9510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:39.958812 9510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46779
I20260808 02:29:39.958848 9586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46779 every 8 connection(s)
I20260808 02:29:39.960280 9510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:39.964258 9587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:39.965576 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 9510
I20260808 02:29:39.965822 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:39.966691 20380 external_mini_cluster.cc:1428] Setting key 0a81a8d2c4c74f0d1bb4e10432a07e41
I20260808 02:29:39.978066 9587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c: Bootstrap starting.
I20260808 02:29:39.981770 9587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:39.982877 9587 log.cc:824] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:29:39 dist-test-slave-pgkr krb5kdc[9494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:29:39.986389 9587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c: No bootstrap required, opened a new log
I20260808 02:29:39.993834 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:39.969043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41929 (local address 127.0.0.1:46779)
0808 02:29:39.970094 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:39.970112 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:39.970494 (+ 382us) server_negotiation.cc:407] Connection header received
0808 02:29:39.972998 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:39.973025 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:39.973538 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:39.974014 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:39.975413 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:39.977410 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:39.978915 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:39.980015 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:39.980210 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:39.984727 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:39.984739 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:39.984757 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:39.984798 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:39.988938 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:39.989779 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:39.989789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:39.989804 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:39.990020 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:39.990458 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:39.990468 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:39.990472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:39.991227 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:39.991563 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:39.991996 (+ 433us) server_negotiation.cc:299] Negotiation successful
0808 02:29:39.992373 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":463,"threads_started":1}
I20260808 02:29:39.997002 9587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798d0882bb4545d88e2e8889ee87ac1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260808 02:29:39.997502 9587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:39.997638 9587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798d0882bb4545d88e2e8889ee87ac1c, State: Initialized, Role: FOLLOWER
I20260808 02:29:39.998566 9587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798d0882bb4545d88e2e8889ee87ac1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260808 02:29:39.998816 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:39.998977 9594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 02:29:39.999327 9594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798d0882bb4545d88e2e8889ee87ac1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260808 02:29:40.000031 9594 leader_election.cc:302] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 798d0882bb4545d88e2e8889ee87ac1c; no voters:
I20260808 02:29:40.000993 9587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:40.001106 9594 leader_election.cc:288] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260808 02:29:40.001241 9587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:40.001407 9587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:40.004958 9587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798d0882bb4545d88e2e8889ee87ac1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260808 02:29:40.005298 9587 leader_election.cc:302] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 798d0882bb4545d88e2e8889ee87ac1c; no voters:
I20260808 02:29:40.005371 9595 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 02:29:40.005556 9587 leader_election.cc:288] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:40.006552 9595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798d0882bb4545d88e2e8889ee87ac1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } } }
I20260808 02:29:40.007205 9595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:29:40.007532 9594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:40.007751 9594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [term 1 LEADER]: Becoming Leader. State: Replica: 798d0882bb4545d88e2e8889ee87ac1c, State: Running, Role: LEADER
I20260808 02:29:40.008595 9587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:40.008505 9594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798d0882bb4545d88e2e8889ee87ac1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260808 02:29:40.011268 9595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798d0882bb4545d88e2e8889ee87ac1c. Latest consensus state: current_term: 1 leader_uuid: "798d0882bb4545d88e2e8889ee87ac1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798d0882bb4545d88e2e8889ee87ac1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } } }
I20260808 02:29:40.011539 9595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:40.012607 9606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:40.034291 9606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:40.040702 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:40.005017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34573 (local address 127.0.0.1:46779)
0808 02:29:40.005225 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:40.005231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:40.011672 (+ 6441us) server_negotiation.cc:407] Connection header received
0808 02:29:40.012243 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:40.012251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:40.012373 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:40.012461 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:40.015068 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:40.016846 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:40.019705 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:40.020795 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:40.020932 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:40.025011 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:40.025018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:40.025027 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:40.025063 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:40.035326 (+ 10263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:40.036612 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:40.036618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:40.036622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:40.037094 (+ 472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:40.037801 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:40.037807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:40.037810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:40.038291 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:40.038714 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:40.040097 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0808 02:29:40.040294 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:29:40.046836 9606 catalog_manager.cc:1347] Generated new cluster ID: a60205cebbb347a6b754e3a5dc9be86f
I20260808 02:29:40.046960 9606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:40.055910 9619 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:29:40.064405 9606 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:40.065837 9606 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:40.080839 9606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 798d0882bb4545d88e2e8889ee87ac1c: Generated new TSK 0
I20260808 02:29:40.081987 9606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:40.097011 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9510
I20260808 02:29:40.099016 9582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:40.195925 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 9510
W20260808 02:29:40.219313 9492 connection.cc:535] client connection to 127.0.0.1:46779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46779): Connection reset by peer (error 104)
2026-08-08T02:29:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:40 dist-test-slave-pgkr krb5kdc[9623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:40 dist-test-slave-pgkr krb5kdc[9623](info): set up 2 sockets
Aug 08 02:29:40 dist-test-slave-pgkr krb5kdc[9623](info): commencing operation
krb5kdc: starting...
W20260808 02:29:41.388113 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:41 dist-test-slave-pgkr krb5kdc[9623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:41.730201 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:42.086177 9636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:42.086475 9636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:42.086719 9636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:42.094251 9636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:42.094372 9636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:42.094463 9636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:42.094573 9636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:42.094700 9636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:42.105283 9636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:42.106705 9636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:42.108129 9636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:42.120688 9641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:42.120752 9642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:42.120738 9644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:42.121858 9636 server_base.cc:1034] running on GCE node
I20260808 02:29:42.122661 9636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:42.124097 9636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:42.125313 9636 hybrid_clock.cc:648] HybridClock initialized: now 1786156182125264 us; error 33 us; skew 500 ppm
I20260808 02:29:42.125636 9636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:42 dist-test-slave-pgkr krb5kdc[9623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:42.128906 9636 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:42.130465 9636 webserver.cc:460] Webserver started at http://127.0.0.1:39389/ using document root <none> and password file <none>
I20260808 02:29:42.131170 9636 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:42.131286 9636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:42.131531 9636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:42.134477 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "2c628b248e5148eb87df66ac6e3229dc"
format_stamp: "Formatted at 2026-08-08 02:29:42 on dist-test-slave-pgkr"
server_key: "0475374dd3a4ddb68b5d137df4e17ea9"
server_key_iv: "9a7d8fd94f47e5c1d8473ee2eb66be8b"
server_key_version: "encryptionkey@0"
I20260808 02:29:42.135289 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "2c628b248e5148eb87df66ac6e3229dc"
format_stamp: "Formatted at 2026-08-08 02:29:42 on dist-test-slave-pgkr"
server_key: "0475374dd3a4ddb68b5d137df4e17ea9"
server_key_iv: "9a7d8fd94f47e5c1d8473ee2eb66be8b"
server_key_version: "encryptionkey@0"
I20260808 02:29:42.140581 9636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:29:42.144150 9652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:42.145530 9636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:29:42.145722 9636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "2c628b248e5148eb87df66ac6e3229dc"
format_stamp: "Formatted at 2026-08-08 02:29:42 on dist-test-slave-pgkr"
server_key: "0475374dd3a4ddb68b5d137df4e17ea9"
server_key_iv: "9a7d8fd94f47e5c1d8473ee2eb66be8b"
server_key_version: "encryptionkey@0"
I20260808 02:29:42.145915 9636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:42.174510 9636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:42.178777 9636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:42.179088 9636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:42.199136 9636 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43331
I20260808 02:29:42.199235 9709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43331 every 8 connection(s)
I20260808 02:29:42.200672 9636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:42.205153 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:42.205514 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 9636
I20260808 02:29:42.205716 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:42.206167 20380 external_mini_cluster.cc:1428] Setting key 2e5f1d67f98ef79ca1773957decb5483
I20260808 02:29:42.219592 9710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc: Bootstrap starting.
I20260808 02:29:42.220122 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:42.207940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40929 (local address 127.0.0.1:43331)
0808 02:29:42.208771 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:42.208783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:42.208828 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:29:42.210511 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:42.210534 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:42.211011 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:42.211430 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:42.212013 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:42.213349 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:42.214734 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:42.215840 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:42.216073 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:42.217312 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:42.217323 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:42.217341 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:42.218032 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:42.218148 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:42.218185 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:42.218345 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:42.218580 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 02:29:42.218782 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":373,"threads_started":1}
I20260808 02:29:42.223651 9710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:42.224334 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:42.224951 9710 log.cc:824] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:42.228264 9710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc: No bootstrap required, opened a new log
I20260808 02:29:42.236572 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:42.229556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54203 (local address 127.0.0.1:43331)
0808 02:29:42.229740 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:42.229747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:42.230093 (+ 346us) server_negotiation.cc:407] Connection header received
0808 02:29:42.230244 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:42.230251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:42.230342 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:42.230473 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:42.231035 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:42.232303 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:42.233616 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:42.234477 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:42.234643 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:42.235771 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:42.235777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:42.235782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:42.235803 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:42.235875 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:42.235896 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:42.236039 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:42.236176 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 02:29:42.236229 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 02:29:42.239029 9710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c628b248e5148eb87df66ac6e3229dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260808 02:29:42.239437 9710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:42.239559 9710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c628b248e5148eb87df66ac6e3229dc, State: Initialized, Role: FOLLOWER
I20260808 02:29:42.240178 9710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c628b248e5148eb87df66ac6e3229dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260808 02:29:42.240427 9710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:42.240590 9710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:42.240764 9710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:42.243144 9710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c628b248e5148eb87df66ac6e3229dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260808 02:29:42.243711 9710 leader_election.cc:302] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c628b248e5148eb87df66ac6e3229dc; no voters:
I20260808 02:29:42.244289 9710 leader_election.cc:288] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:42.244472 9720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:42.245240 9720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [term 1 LEADER]: Becoming Leader. State: Replica: 2c628b248e5148eb87df66ac6e3229dc, State: Running, Role: LEADER
I20260808 02:29:42.245776 9720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c628b248e5148eb87df66ac6e3229dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } }
I20260808 02:29:42.246510 9710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:42.248708 9723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c628b248e5148eb87df66ac6e3229dc. Latest consensus state: current_term: 1 leader_uuid: "2c628b248e5148eb87df66ac6e3229dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c628b248e5148eb87df66ac6e3229dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } } }
I20260808 02:29:42.249853 9723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:42.249212 9721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c628b248e5148eb87df66ac6e3229dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c628b248e5148eb87df66ac6e3229dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43331 } } }
I20260808 02:29:42.251420 9721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:42.252681 9726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:42.258170 9726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:42.268672 9726 catalog_manager.cc:1347] Generated new cluster ID: 79ff149f1d63488993dd5042ba4df8fd
I20260808 02:29:42.268941 9726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:42.286115 9726 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:42.287621 9726 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:42.295460 9726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c628b248e5148eb87df66ac6e3229dc: Generated new TSK 0
I20260808 02:29:42.296386 9726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:42.314886 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9636
W20260808 02:29:42.316104 9704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40929
W20260808 02:29:42.317061 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9636: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:29:42.317265 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 9636
W20260808 02:29:42.340409 9621 connection.cc:535] client connection to 127.0.0.1:43331 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43331): Connection reset by peer (error 104)
2026-08-08T02:29:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2120 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:42 dist-test-slave-pgkr krb5kdc[9744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:42 dist-test-slave-pgkr krb5kdc[9744](info): set up 2 sockets
Aug 08 02:29:42 dist-test-slave-pgkr krb5kdc[9744](info): commencing operation
krb5kdc: starting...
W20260808 02:29:43.540179 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.003s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:43 dist-test-slave-pgkr krb5kdc[9744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:43.884018 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:44.253459 9758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:44.253785 9758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:44.253979 9758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:44.261229 9758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:44.261365 9758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:44.261456 9758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:44.261580 9758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:44.261665 9758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:44.271612 9758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:44.272994 9758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:44.274405 9758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:44.287552 9763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:44.287575 9764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:44.287635 9766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:44.288513 9758 server_base.cc:1034] running on GCE node
I20260808 02:29:44.289450 9758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:44.291024 9758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:44.292294 9758 hybrid_clock.cc:648] HybridClock initialized: now 1786156184292246 us; error 48 us; skew 500 ppm
I20260808 02:29:44.292692 9758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:44 dist-test-slave-pgkr krb5kdc[9744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:44.296144 9758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:44.297729 9758 webserver.cc:460] Webserver started at http://127.0.0.1:42517/ using document root <none> and password file <none>
I20260808 02:29:44.298471 9758 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:44.298668 9758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:44.299010 9758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:44.302143 9758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "841eb50107c94fcdbe618798caf0070f"
format_stamp: "Formatted at 2026-08-08 02:29:44 on dist-test-slave-pgkr"
server_key: "29387ecc4b8f85831267e238c9b6b3d7"
server_key_iv: "22cb30e8ef7c26ee54b37af6b8d249dd"
server_key_version: "encryptionkey@0"
I20260808 02:29:44.303017 9758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "841eb50107c94fcdbe618798caf0070f"
format_stamp: "Formatted at 2026-08-08 02:29:44 on dist-test-slave-pgkr"
server_key: "29387ecc4b8f85831267e238c9b6b3d7"
server_key_iv: "22cb30e8ef7c26ee54b37af6b8d249dd"
server_key_version: "encryptionkey@0"
I20260808 02:29:44.308437 9758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260808 02:29:44.311990 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:44.313362 9758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:29:44.313555 9758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "841eb50107c94fcdbe618798caf0070f"
format_stamp: "Formatted at 2026-08-08 02:29:44 on dist-test-slave-pgkr"
server_key: "29387ecc4b8f85831267e238c9b6b3d7"
server_key_iv: "22cb30e8ef7c26ee54b37af6b8d249dd"
server_key_version: "encryptionkey@0"
I20260808 02:29:44.313755 9758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:44.319597 9758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:44.323990 9758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:44.324309 9758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:44.343716 9758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33555
I20260808 02:29:44.343770 9831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33555 every 8 connection(s)
I20260808 02:29:44.345214 9758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:44.349058 9832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:44.349139 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 9758
I20260808 02:29:44.349337 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:44.349792 20380 external_mini_cluster.cc:1428] Setting key 031254e661a5afa9384dc812e39c99fd
I20260808 02:29:44.357818 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:44.351593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37421 (local address 127.0.0.1:33555)
0808 02:29:44.352488 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:44.352499 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:44.352542 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:29:44.354180 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:44.354203 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:44.354560 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:44.355027 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:44.356114 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:44.356120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:44.356132 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:44.356149 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:44.356204 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:44.356222 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:44.356467 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:44.356614 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 02:29:44.356818 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":374,"threads_started":1}
I20260808 02:29:44.362119 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:44.363567 9832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f: Bootstrap starting.
I20260808 02:29:44.367857 9832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:44.369407 9832 log.cc:824] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:44.370560 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:44.367792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36101 (local address 127.0.0.1:33555)
0808 02:29:44.368056 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:44.368063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:44.368378 (+ 315us) server_negotiation.cc:407] Connection header received
0808 02:29:44.368578 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:44.368582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:44.368634 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:44.368727 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:44.369757 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:44.369761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:44.369764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:44.369781 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:44.369820 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:44.369834 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:44.369952 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:44.370130 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 02:29:44.370205 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 02:29:44.372766 9832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f: No bootstrap required, opened a new log
I20260808 02:29:44.380470 9832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841eb50107c94fcdbe618798caf0070f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260808 02:29:44.380879 9832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:44.381000 9832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841eb50107c94fcdbe618798caf0070f, State: Initialized, Role: FOLLOWER
I20260808 02:29:44.381644 9832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841eb50107c94fcdbe618798caf0070f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260808 02:29:44.381893 9832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:44.382061 9832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:44.382234 9832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:44.384933 9832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841eb50107c94fcdbe618798caf0070f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260808 02:29:44.385468 9832 leader_election.cc:302] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 841eb50107c94fcdbe618798caf0070f; no voters:
I20260808 02:29:44.386164 9832 leader_election.cc:288] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:44.386301 9842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:44.387768 9842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [term 1 LEADER]: Becoming Leader. State: Replica: 841eb50107c94fcdbe618798caf0070f, State: Running, Role: LEADER
I20260808 02:29:44.388374 9842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841eb50107c94fcdbe618798caf0070f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } }
I20260808 02:29:44.388571 9832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:44.391867 9844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841eb50107c94fcdbe618798caf0070f. Latest consensus state: current_term: 1 leader_uuid: "841eb50107c94fcdbe618798caf0070f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841eb50107c94fcdbe618798caf0070f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } } }
I20260808 02:29:44.391839 9843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "841eb50107c94fcdbe618798caf0070f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841eb50107c94fcdbe618798caf0070f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33555 } } }
I20260808 02:29:44.392392 9843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:44.392367 9844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:44.395123 9851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:44.400643 9851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:44.410233 9851 catalog_manager.cc:1347] Generated new cluster ID: 4a73223fa73c4dc793df2a3cdba27450
I20260808 02:29:44.410359 9851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:44.426080 9851 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:44.426532 9851 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:44.432336 9851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 841eb50107c94fcdbe618798caf0070f: Generated new TSK 0
I20260808 02:29:44.433122 9851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:44.442078 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9758
W20260808 02:29:44.443506 9827 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37421
W20260808 02:29:44.444234 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9758: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:29:44.444399 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 9758
2026-08-08T02:29:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:44 dist-test-slave-pgkr krb5kdc[9863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:44 dist-test-slave-pgkr krb5kdc[9863](info): set up 2 sockets
Aug 08 02:29:44 dist-test-slave-pgkr krb5kdc[9863](info): commencing operation
krb5kdc: starting...
W20260808 02:29:45.631196 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.007s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:45 dist-test-slave-pgkr krb5kdc[9863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:45.985473 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:46.350409 9879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:46.350841 9879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:46.350960 9879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:46.358382 9879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:46.358492 9879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:46.358634 9879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:46.358726 9879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:46.358774 9879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:46.369339 9879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:46.370748 9879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:46.372100 9879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:46.384277 9884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:46.384367 9885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:46.384377 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:46.385322 9879 server_base.cc:1034] running on GCE node
I20260808 02:29:46.386193 9879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:46.387703 9879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:46.388954 9879 hybrid_clock.cc:648] HybridClock initialized: now 1786156186388892 us; error 52 us; skew 500 ppm
I20260808 02:29:46.389331 9879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:46 dist-test-slave-pgkr krb5kdc[9863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:46.396379 9879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:46.398069 9879 webserver.cc:460] Webserver started at http://127.0.0.1:43491/ using document root <none> and password file <none>
I20260808 02:29:46.398861 9879 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:46.399008 9879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:46.399333 9879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:46.402509 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "6a73755e152540dca3494d0b85f8e934"
format_stamp: "Formatted at 2026-08-08 02:29:46 on dist-test-slave-pgkr"
server_key: "7cb0846132f4ebaaa4ee55e1505da1f5"
server_key_iv: "4daeded96712e6f9b20087f422deaa80"
server_key_version: "encryptionkey@0"
I20260808 02:29:46.403378 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "6a73755e152540dca3494d0b85f8e934"
format_stamp: "Formatted at 2026-08-08 02:29:46 on dist-test-slave-pgkr"
server_key: "7cb0846132f4ebaaa4ee55e1505da1f5"
server_key_iv: "4daeded96712e6f9b20087f422deaa80"
server_key_version: "encryptionkey@0"
I20260808 02:29:46.408718 9879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 02:29:46.412487 9896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:46.413883 9879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:29:46.414074 9879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "6a73755e152540dca3494d0b85f8e934"
format_stamp: "Formatted at 2026-08-08 02:29:46 on dist-test-slave-pgkr"
server_key: "7cb0846132f4ebaaa4ee55e1505da1f5"
server_key_iv: "4daeded96712e6f9b20087f422deaa80"
server_key_version: "encryptionkey@0"
I20260808 02:29:46.414279 9879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:46.434669 9879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:46.439049 9879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:46.439384 9879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:46.464104 9879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37957
I20260808 02:29:46.464123 9950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37957 every 8 connection(s)
I20260808 02:29:46.465545 9879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:46.469383 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:46.470745 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 9879
I20260808 02:29:46.470945 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:46.471407 20380 external_mini_cluster.cc:1428] Setting key 569aae4b18dec1808ec47fcb7a778bdf
I20260808 02:29:46.483970 9951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934: Bootstrap starting.
Aug 08 02:29:46 dist-test-slave-pgkr krb5kdc[9863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:29:46.487560 9951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:46.488749 9951 log.cc:824] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:46.492132 9951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934: No bootstrap required, opened a new log
I20260808 02:29:46.495956 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:46.473367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33797 (local address 127.0.0.1:37957)
0808 02:29:46.474209 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:46.474224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:46.474277 (+ 53us) server_negotiation.cc:407] Connection header received
0808 02:29:46.476193 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:46.476224 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:46.476778 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:46.477225 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:46.478670 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:46.480748 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:46.482368 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:46.483444 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:46.483640 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:46.488078 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:46.488086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:46.488101 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:46.488137 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:46.491677 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:46.492512 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:46.492519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:46.492525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:46.492714 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:46.493129 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:46.493134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:46.493137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:46.493667 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:46.494016 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:46.494483 (+ 467us) server_negotiation.cc:299] Negotiation successful
0808 02:29:46.494945 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":372,"threads_started":1}
I20260808 02:29:46.499958 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:46.501137 9951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a73755e152540dca3494d0b85f8e934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260808 02:29:46.501504 9951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:46.501618 9951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a73755e152540dca3494d0b85f8e934, State: Initialized, Role: FOLLOWER
I20260808 02:29:46.502173 9951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a73755e152540dca3494d0b85f8e934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260808 02:29:46.502475 9951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:46.502704 9951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:46.502971 9951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:46.506083 9951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a73755e152540dca3494d0b85f8e934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260808 02:29:46.506587 9951 leader_election.cc:302] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a73755e152540dca3494d0b85f8e934; no voters:
I20260808 02:29:46.507351 9951 leader_election.cc:288] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:46.507453 9964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:46.509222 9964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [term 1 LEADER]: Becoming Leader. State: Replica: 6a73755e152540dca3494d0b85f8e934, State: Running, Role: LEADER
I20260808 02:29:46.509888 9951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:46.509857 9964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a73755e152540dca3494d0b85f8e934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260808 02:29:46.515529 9965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a73755e152540dca3494d0b85f8e934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a73755e152540dca3494d0b85f8e934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260808 02:29:46.516062 9965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:46.516664 9966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a73755e152540dca3494d0b85f8e934. Latest consensus state: current_term: 1 leader_uuid: "6a73755e152540dca3494d0b85f8e934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a73755e152540dca3494d0b85f8e934" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260808 02:29:46.517021 9966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:46.520401 9970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:46.523240 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:46.505024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44775 (local address 127.0.0.1:37957)
0808 02:29:46.505195 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:46.505202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:46.505560 (+ 358us) server_negotiation.cc:407] Connection header received
0808 02:29:46.505768 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:46.505772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:46.505829 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:46.505934 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:46.507734 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:46.508823 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:46.510372 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:46.510954 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:46.511097 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:46.513809 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:46.513819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:46.513826 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:46.513877 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:46.518526 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:46.519856 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:46.519865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:46.519871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:46.520078 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:46.520634 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:46.520638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:46.520642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:46.521194 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:46.522316 (+ 1122us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:46.522507 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 02:29:46.522815 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 02:29:46.527316 9970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:46.538446 9970 catalog_manager.cc:1347] Generated new cluster ID: 18cb3d349cad45f493305f0fb72e224c
I20260808 02:29:46.538671 9970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:46.562479 9970 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:46.564404 9970 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:46.576470 9970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a73755e152540dca3494d0b85f8e934: Generated new TSK 0
I20260808 02:29:46.577574 9970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:46.591463 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 9879
I20260808 02:29:46.592653 9946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:46.691486 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 9879
2026-08-08T02:29:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:46 dist-test-slave-pgkr krb5kdc[9988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:46 dist-test-slave-pgkr krb5kdc[9988](info): set up 2 sockets
Aug 08 02:29:46 dist-test-slave-pgkr krb5kdc[9988](info): commencing operation
krb5kdc: starting...
W20260808 02:29:47.871590 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:48 dist-test-slave-pgkr krb5kdc[9988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:48.230569 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:48.596901 10001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:48.597239 10001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:48.597441 10001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:48.605777 10001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:48.605937 10001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:48.606046 10001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:48.606163 10001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:48.606252 10001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:48.617666 10001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:48.619187 10001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:48.620623 10001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:48.633097 10007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:48.633165 10008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:48.633555 10010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:48.633985 10001 server_base.cc:1034] running on GCE node
I20260808 02:29:48.634858 10001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:48.636334 10001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:48.637548 10001 hybrid_clock.cc:648] HybridClock initialized: now 1786156188637478 us; error 82 us; skew 500 ppm
I20260808 02:29:48.637975 10001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:48 dist-test-slave-pgkr krb5kdc[9988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:48.641319 10001 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:48.642940 10001 webserver.cc:460] Webserver started at http://127.0.0.1:45995/ using document root <none> and password file <none>
I20260808 02:29:48.643638 10001 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:48.643808 10001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:48.644127 10001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:48.647254 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "82705601d36b468289e07797893518b6"
format_stamp: "Formatted at 2026-08-08 02:29:48 on dist-test-slave-pgkr"
server_key: "5ba09a85dc4393a63d83447f23910cab"
server_key_iv: "e8339c73a59a335d6aacf69439c49733"
server_key_version: "encryptionkey@0"
I20260808 02:29:48.648082 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "82705601d36b468289e07797893518b6"
format_stamp: "Formatted at 2026-08-08 02:29:48 on dist-test-slave-pgkr"
server_key: "5ba09a85dc4393a63d83447f23910cab"
server_key_iv: "e8339c73a59a335d6aacf69439c49733"
server_key_version: "encryptionkey@0"
I20260808 02:29:48.653434 10001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 02:29:48.657012 10018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:48.658349 10001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:29:48.658545 10001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "82705601d36b468289e07797893518b6"
format_stamp: "Formatted at 2026-08-08 02:29:48 on dist-test-slave-pgkr"
server_key: "5ba09a85dc4393a63d83447f23910cab"
server_key_iv: "e8339c73a59a335d6aacf69439c49733"
server_key_version: "encryptionkey@0"
I20260808 02:29:48.658799 10001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:48.675431 10001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:48.679771 10001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:48.680143 10001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:48.700335 10001 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44403
I20260808 02:29:48.700366 10072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44403 every 8 connection(s)
I20260808 02:29:48.701794 10001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:48.705446 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10001
I20260808 02:29:48.705651 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:48.706092 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:48.706252 20380 external_mini_cluster.cc:1428] Setting key 718ab0aff669b98c17a96e5509bb2681
I20260808 02:29:48.720467 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:48.708174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60065 (local address 127.0.0.1:44403)
0808 02:29:48.709083 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:48.709099 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:48.709142 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:29:48.711112 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:48.711134 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:48.711556 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:48.711901 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:48.712536 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:48.713899 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:48.715759 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:48.716580 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:48.716725 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:48.717977 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:48.717984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:48.717998 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:48.718569 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:48.718711 (+ 142us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:48.718737 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:48.718825 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:48.719116 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 02:29:48.719422 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":438,"threads_started":1}
I20260808 02:29:48.720510 10073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6: Bootstrap starting.
I20260808 02:29:48.724004 10073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:48.724630 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:48.725311 10073 log.cc:824] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:48.728192 10073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6: No bootstrap required, opened a new log
I20260808 02:29:48.736433 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:48.729442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39869 (local address 127.0.0.1:44403)
0808 02:29:48.729647 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:48.729656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:48.730214 (+ 558us) server_negotiation.cc:407] Connection header received
0808 02:29:48.730380 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:48.730386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:48.730472 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:48.730576 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:48.731175 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:48.732317 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:48.733645 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:48.734375 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:48.734508 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:48.735633 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:48.735639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:48.735644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:48.735664 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:48.735721 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:48.735741 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:48.735842 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:48.735978 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 02:29:48.736070 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:29:48.737805 10073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82705601d36b468289e07797893518b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44403 } }
I20260808 02:29:48.738216 10073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:48.738328 10073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82705601d36b468289e07797893518b6, State: Initialized, Role: FOLLOWER
I20260808 02:29:48.738963 10073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82705601d36b468289e07797893518b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44403 } }
I20260808 02:29:48.739230 10073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:48.739384 10073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:48.739575 10073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:48.741782 10073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82705601d36b468289e07797893518b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44403 } }
I20260808 02:29:48.742306 10073 leader_election.cc:302] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82705601d36b468289e07797893518b6; no voters:
I20260808 02:29:48.742990 10073 leader_election.cc:288] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:48.743113 10084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:48.743978 10084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [term 1 LEADER]: Becoming Leader. State: Replica: 82705601d36b468289e07797893518b6, State: Running, Role: LEADER
I20260808 02:29:48.744556 10084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82705601d36b468289e07797893518b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44403 } }
I20260808 02:29:48.745265 10073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:48.747309 10085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82705601d36b468289e07797893518b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82705601d36b468289e07797893518b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44403 } } }
I20260808 02:29:48.747643 10085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:48.750344 10090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:48.750190 10086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82705601d36b468289e07797893518b6. Latest consensus state: current_term: 1 leader_uuid: "82705601d36b468289e07797893518b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82705601d36b468289e07797893518b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44403 } } }
I20260808 02:29:48.750535 10086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:48.755852 10090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:48.767817 10090 catalog_manager.cc:1347] Generated new cluster ID: 596cf4f51c0b4e98bf93229c3f53e8e2
I20260808 02:29:48.768014 10090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:48.817386 10090 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:48.818835 10090 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:48.829444 10090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82705601d36b468289e07797893518b6: Generated new TSK 0
I20260808 02:29:48.830322 10090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:48.843590 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10001
W20260808 02:29:48.844746 10068 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60065
W20260808 02:29:48.845626 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:29:48.845809 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10001
2026-08-08T02:29:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2155 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:48 dist-test-slave-pgkr krb5kdc[10106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:48 dist-test-slave-pgkr krb5kdc[10106](info): set up 2 sockets
Aug 08 02:29:48 dist-test-slave-pgkr krb5kdc[10106](info): commencing operation
krb5kdc: starting...
W20260808 02:29:50.031442 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:50 dist-test-slave-pgkr krb5kdc[10106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:50.381986 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:50.737322 10121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:50.737700 10121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:50.737871 10121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:50.745112 10121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:50.745280 10121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:50.745393 10121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:50.745508 10121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:50.745599 10121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:50.755614 10121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:50.756985 10121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:50.758311 10121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:50.770428 10127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:50.770485 10130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:50.770517 10128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:50.771706 10121 server_base.cc:1034] running on GCE node
I20260808 02:29:50.772470 10121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:50.773933 10121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:50.775211 10121 hybrid_clock.cc:648] HybridClock initialized: now 1786156190775154 us; error 46 us; skew 500 ppm
I20260808 02:29:50.775595 10121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:50 dist-test-slave-pgkr krb5kdc[10106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:50.778915 10121 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:50.780441 10121 webserver.cc:460] Webserver started at http://127.0.0.1:45743/ using document root <none> and password file <none>
I20260808 02:29:50.781169 10121 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:50.781337 10121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:50.781672 10121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:50.784794 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "52c2579027e14acb940a09364adfdd60"
format_stamp: "Formatted at 2026-08-08 02:29:50 on dist-test-slave-pgkr"
server_key: "fb8415d95b1d659ac6c5936ff3e6702f"
server_key_iv: "95e283d2b1e757c25bb814d5394f1fa7"
server_key_version: "encryptionkey@0"
I20260808 02:29:50.785580 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "52c2579027e14acb940a09364adfdd60"
format_stamp: "Formatted at 2026-08-08 02:29:50 on dist-test-slave-pgkr"
server_key: "fb8415d95b1d659ac6c5936ff3e6702f"
server_key_iv: "95e283d2b1e757c25bb814d5394f1fa7"
server_key_version: "encryptionkey@0"
I20260808 02:29:50.790982 10121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 02:29:50.794474 10137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:50.795866 10121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:50.796159 10121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "52c2579027e14acb940a09364adfdd60"
format_stamp: "Formatted at 2026-08-08 02:29:50 on dist-test-slave-pgkr"
server_key: "fb8415d95b1d659ac6c5936ff3e6702f"
server_key_iv: "95e283d2b1e757c25bb814d5394f1fa7"
server_key_version: "encryptionkey@0"
I20260808 02:29:50.796380 10121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:50.802299 10121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:50.806802 10121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:50.807112 10121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:50.827117 10121 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44319
I20260808 02:29:50.827148 10194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44319 every 8 connection(s)
I20260808 02:29:50.828572 10121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:50.832608 10195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:50.837508 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10121
I20260808 02:29:50.837702 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:50.838168 20380 external_mini_cluster.cc:1428] Setting key d1ae3ff371374fb0ecefb945d9cc5a05
I20260808 02:29:50.843801 10195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60: Bootstrap starting.
I20260808 02:29:50.846423 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:50.840124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39257 (local address 127.0.0.1:44319)
0808 02:29:50.841036 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:50.841048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:50.841113 (+ 65us) server_negotiation.cc:407] Connection header received
0808 02:29:50.842932 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:50.842956 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:50.843306 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:50.843624 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:50.844768 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:50.844774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:50.844787 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:50.844801 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:50.844862 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:50.844887 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:50.845052 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:50.845213 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 02:29:50.845482 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":438,"threads_started":1}
I20260808 02:29:50.847589 10195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:50.848757 10195 log.cc:824] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:50.850445 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:50.852344 10195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60: No bootstrap required, opened a new log
I20260808 02:29:50.858301 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:50.855747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33931 (local address 127.0.0.1:44319)
0808 02:29:50.855982 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:50.855989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:50.856351 (+ 362us) server_negotiation.cc:407] Connection header received
0808 02:29:50.856513 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:50.856517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:50.856563 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:50.856645 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:50.857610 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:50.857614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:50.857617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:50.857630 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:50.857672 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:50.857687 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:50.857774 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:50.857914 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 02:29:50.857964 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:29:50.862138 10195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c2579027e14acb940a09364adfdd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260808 02:29:50.862540 10195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:50.862701 10195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c2579027e14acb940a09364adfdd60, State: Initialized, Role: FOLLOWER
I20260808 02:29:50.863323 10195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c2579027e14acb940a09364adfdd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260808 02:29:50.863548 10195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:50.863648 10195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:50.863798 10195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:50.866381 10195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c2579027e14acb940a09364adfdd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260808 02:29:50.866906 10195 leader_election.cc:302] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c2579027e14acb940a09364adfdd60; no voters:
I20260808 02:29:50.867456 10195 leader_election.cc:288] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:50.867602 10206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:50.868381 10206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [term 1 LEADER]: Becoming Leader. State: Replica: 52c2579027e14acb940a09364adfdd60, State: Running, Role: LEADER
I20260808 02:29:50.868911 10206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c2579027e14acb940a09364adfdd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } }
I20260808 02:29:50.869807 10195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:50.872169 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c2579027e14acb940a09364adfdd60. Latest consensus state: current_term: 1 leader_uuid: "52c2579027e14acb940a09364adfdd60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c2579027e14acb940a09364adfdd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } } }
I20260808 02:29:50.872581 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:50.872289 10207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c2579027e14acb940a09364adfdd60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c2579027e14acb940a09364adfdd60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44319 } } }
I20260808 02:29:50.872766 10207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:50.876189 10213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:50.881843 10213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:50.890491 10213 catalog_manager.cc:1347] Generated new cluster ID: 05655262d1ce46e0af6b342f1f62224c
I20260808 02:29:50.890623 10213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:50.937601 10213 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:50.938076 10213 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:50.963280 10213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c2579027e14acb940a09364adfdd60: Generated new TSK 0
I20260808 02:29:50.964411 10213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:50.983773 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10121
W20260808 02:29:50.985196 10189 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39257
W20260808 02:29:50.986023 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10121: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:29:50.986191 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10121
2026-08-08T02:29:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2140 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:51 dist-test-slave-pgkr krb5kdc[10228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:51 dist-test-slave-pgkr krb5kdc[10228](info): set up 2 sockets
Aug 08 02:29:51 dist-test-slave-pgkr krb5kdc[10228](info): commencing operation
krb5kdc: starting...
W20260808 02:29:52.214401 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:52 dist-test-slave-pgkr krb5kdc[10228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:52.567574 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:52.937461 10243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:52.937775 10243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:52.937973 10243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:52.946012 10243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:52.946156 10243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:52.946240 10243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:52.946326 10243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:52.946408 10243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:52.956451 10243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:52.957804 10243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:52.959218 10243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:52.970973 10249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:52.971055 10253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:52.971081 10250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:52.972222 10243 server_base.cc:1034] running on GCE node
I20260808 02:29:52.973052 10243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:52.974466 10243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:52.975728 10243 hybrid_clock.cc:648] HybridClock initialized: now 1786156192975674 us; error 67 us; skew 500 ppm
I20260808 02:29:52.976096 10243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:52 dist-test-slave-pgkr krb5kdc[10228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:52.984606 10243 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:52.986372 10243 webserver.cc:460] Webserver started at http://127.0.0.1:35533/ using document root <none> and password file <none>
I20260808 02:29:52.987201 10243 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:52.987371 10243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:52.987730 10243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:52.991252 10243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "9f1f6172019d4b19a6cb741f94a55797"
format_stamp: "Formatted at 2026-08-08 02:29:52 on dist-test-slave-pgkr"
server_key: "3f2f21d0876190e030c9218153a1dc5d"
server_key_iv: "746369421fe7f402fb2a4da22370f7b0"
server_key_version: "encryptionkey@0"
I20260808 02:29:52.992132 10243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "9f1f6172019d4b19a6cb741f94a55797"
format_stamp: "Formatted at 2026-08-08 02:29:52 on dist-test-slave-pgkr"
server_key: "3f2f21d0876190e030c9218153a1dc5d"
server_key_iv: "746369421fe7f402fb2a4da22370f7b0"
server_key_version: "encryptionkey@0"
I20260808 02:29:52.998159 10243 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260808 02:29:53.002314 10260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:53.003834 10243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:53.004050 10243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "9f1f6172019d4b19a6cb741f94a55797"
format_stamp: "Formatted at 2026-08-08 02:29:52 on dist-test-slave-pgkr"
server_key: "3f2f21d0876190e030c9218153a1dc5d"
server_key_iv: "746369421fe7f402fb2a4da22370f7b0"
server_key_version: "encryptionkey@0"
I20260808 02:29:53.004297 10243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:53.026768 10243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:53.031100 10243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:53.031406 10243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:53.051311 10243 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41369
I20260808 02:29:53.051340 10317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41369 every 8 connection(s)
I20260808 02:29:53.053062 10243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:53.057056 10318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:53.061870 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10243
I20260808 02:29:53.062057 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:53.062502 20380 external_mini_cluster.cc:1428] Setting key 15050bfaad4bbaca1ae30bab798bf677
I20260808 02:29:53.068459 10318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797: Bootstrap starting.
I20260808 02:29:53.072388 10318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:53.073678 10318 log.cc:824] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:53.076757 10318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797: No bootstrap required, opened a new log
Aug 08 02:29:53 dist-test-slave-pgkr krb5kdc[10228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:29:53.086274 10318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f6172019d4b19a6cb741f94a55797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260808 02:29:53.086776 10318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:53.086946 10318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1f6172019d4b19a6cb741f94a55797, State: Initialized, Role: FOLLOWER
I20260808 02:29:53.087255 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:53.064426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:40431 (local address 127.0.0.1:41369)
0808 02:29:53.065243 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:53.065255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:53.065297 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:29:53.067138 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:53.067162 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:53.067571 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:53.067938 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:53.069863 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:53.071310 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:53.073189 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:53.073982 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:53.074135 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:53.079062 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:53.079070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:53.079085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:53.079129 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:53.082521 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:53.083560 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:53.083567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:53.083573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:53.083743 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:53.084273 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:53.084276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:53.084278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:53.084809 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:53.085042 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:53.085569 (+ 527us) server_negotiation.cc:299] Negotiation successful
0808 02:29:53.085881 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":366,"threads_started":1}
I20260808 02:29:53.087658 10318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f6172019d4b19a6cb741f94a55797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260808 02:29:53.087949 10318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:53.088109 10318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:53.088308 10318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:53.091655 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:53.091670 10318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f6172019d4b19a6cb741f94a55797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260808 02:29:53.092218 10318 leader_election.cc:302] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1f6172019d4b19a6cb741f94a55797; no voters:
I20260808 02:29:53.092907 10318 leader_election.cc:288] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:53.093053 10323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:53.095007 10318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:53.103713 10323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1f6172019d4b19a6cb741f94a55797, State: Running, Role: LEADER
I20260808 02:29:53.104585 10323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f6172019d4b19a6cb741f94a55797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260808 02:29:53.108373 10325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f1f6172019d4b19a6cb741f94a55797. Latest consensus state: current_term: 1 leader_uuid: "9f1f6172019d4b19a6cb741f94a55797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f6172019d4b19a6cb741f94a55797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260808 02:29:53.108474 10324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f1f6172019d4b19a6cb741f94a55797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f6172019d4b19a6cb741f94a55797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260808 02:29:53.108176 10346 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:29:53.109341 10324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:53.109338 10325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:53.110220 10347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:53.115908 10347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:53.124253 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:53.105582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:50495 (local address 127.0.0.1:41369)
0808 02:29:53.105778 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:53.105786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:53.106266 (+ 480us) server_negotiation.cc:407] Connection header received
0808 02:29:53.106454 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:53.106463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:53.106584 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:53.106732 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:53.109013 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:53.110259 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:53.111964 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:53.112747 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:53.112917 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:53.116302 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:53.116309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:53.116314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:53.116355 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:53.120520 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:53.121527 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:53.121533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:53.121537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:53.121680 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:53.122227 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:53.122232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:53.122236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:53.122607 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:53.122936 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:53.123479 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 02:29:53.123732 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:29:53.126664 10347 catalog_manager.cc:1347] Generated new cluster ID: ae0312c984834c0b8d4dd50ea72b4968
I20260808 02:29:53.126808 10347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:53.144403 10347 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:53.145709 10347 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:53.157370 10347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f1f6172019d4b19a6cb741f94a55797: Generated new TSK 0
I20260808 02:29:53.158461 10347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:53.166476 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10243
I20260808 02:29:53.167748 10313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:53.267283 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10243
W20260808 02:29:53.291178 10226 connection.cc:535] client connection to 127.0.0.1:41369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41369): Connection reset by peer (error 104)
2026-08-08T02:29:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:53 dist-test-slave-pgkr krb5kdc[10354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:53 dist-test-slave-pgkr krb5kdc[10354](info): set up 2 sockets
Aug 08 02:29:53 dist-test-slave-pgkr krb5kdc[10354](info): commencing operation
krb5kdc: starting...
W20260808 02:29:54.507575 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:54 dist-test-slave-pgkr krb5kdc[10354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:54.855808 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:55.214970 10368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:55.215324 10368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:55.215494 10368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:55.222882 10368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:55.223037 10368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:55.223136 10368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:55.223260 10368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:55.223341 10368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:55.233745 10368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:55.235232 10368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:55.236675 10368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:55.249194 10373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:55.249238 10374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:55.249464 10376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:55.252272 10368 server_base.cc:1034] running on GCE node
I20260808 02:29:55.253489 10368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:55.255718 10368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:55.257012 10368 hybrid_clock.cc:648] HybridClock initialized: now 1786156195256896 us; error 105 us; skew 500 ppm
I20260808 02:29:55.257457 10368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:55 dist-test-slave-pgkr krb5kdc[10354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:55.261482 10368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:55.263265 10368 webserver.cc:460] Webserver started at http://127.0.0.1:45063/ using document root <none> and password file <none>
I20260808 02:29:55.264088 10368 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:55.264240 10368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:55.264614 10368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:55.268177 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "4c77e3533d454f9092bef2ed7a532dcd"
format_stamp: "Formatted at 2026-08-08 02:29:55 on dist-test-slave-pgkr"
server_key: "9861393631baffc43bd4a260d0872b59"
server_key_iv: "4d49ad01e4a6654087cad9743a121de9"
server_key_version: "encryptionkey@0"
I20260808 02:29:55.269090 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "4c77e3533d454f9092bef2ed7a532dcd"
format_stamp: "Formatted at 2026-08-08 02:29:55 on dist-test-slave-pgkr"
server_key: "9861393631baffc43bd4a260d0872b59"
server_key_iv: "4d49ad01e4a6654087cad9743a121de9"
server_key_version: "encryptionkey@0"
I20260808 02:29:55.275188 10368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 02:29:55.279142 10383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:55.280658 10368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 02:29:55.280870 10368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "4c77e3533d454f9092bef2ed7a532dcd"
format_stamp: "Formatted at 2026-08-08 02:29:55 on dist-test-slave-pgkr"
server_key: "9861393631baffc43bd4a260d0872b59"
server_key_iv: "4d49ad01e4a6654087cad9743a121de9"
server_key_version: "encryptionkey@0"
I20260808 02:29:55.281060 10368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:55.300130 10368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:55.305116 10368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:55.305459 10368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:55.326390 10368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43973
I20260808 02:29:55.326411 10439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43973 every 8 connection(s)
I20260808 02:29:55.327927 10368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:55.332223 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10368
I20260808 02:29:55.332448 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:55.332562 10440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:55.333020 20380 external_mini_cluster.cc:1428] Setting key b24b131c1b90d5ee11fe884afaad0173
I20260808 02:29:55.348451 10440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd: Bootstrap starting.
I20260808 02:29:55.349085 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:55.335162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:46417 (local address 127.0.0.1:43973)
0808 02:29:55.336085 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:55.336101 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:55.336157 (+ 56us) server_negotiation.cc:407] Connection header received
0808 02:29:55.338121 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:55.338157 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:55.338726 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:55.339185 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:55.339744 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:55.341776 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:55.343422 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:55.344524 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:55.344723 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:55.346101 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:55.346113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:55.346133 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:55.346881 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:55.346990 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:55.347021 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:55.347191 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:55.347516 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 02:29:55.347708 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":428,"threads_started":1}
I20260808 02:29:55.352185 10440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:55.353293 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:55.353510 10440 log.cc:824] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:55.356410 10440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd: No bootstrap required, opened a new log
I20260808 02:29:55.365473 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:55.358010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:45485 (local address 127.0.0.1:43973)
0808 02:29:55.358266 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:55.358272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:55.358715 (+ 443us) server_negotiation.cc:407] Connection header received
0808 02:29:55.358916 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:55.358920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:55.358986 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:55.359074 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:55.359705 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:55.360504 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:55.362442 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:55.363094 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:55.363216 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:55.364481 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:55.364486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:55.364489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:55.364507 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:55.364576 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:55.364592 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:55.364695 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:55.365017 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 02:29:55.365088 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 02:29:55.366496 10440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77e3533d454f9092bef2ed7a532dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260808 02:29:55.366984 10440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:55.367098 10440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c77e3533d454f9092bef2ed7a532dcd, State: Initialized, Role: FOLLOWER
I20260808 02:29:55.367736 10440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77e3533d454f9092bef2ed7a532dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260808 02:29:55.367995 10440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:55.368147 10440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:55.368337 10440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:55.370514 10440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77e3533d454f9092bef2ed7a532dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260808 02:29:55.371052 10440 leader_election.cc:302] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c77e3533d454f9092bef2ed7a532dcd; no voters:
I20260808 02:29:55.371661 10440 leader_election.cc:288] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:55.371835 10453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:55.373310 10453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [term 1 LEADER]: Becoming Leader. State: Replica: 4c77e3533d454f9092bef2ed7a532dcd, State: Running, Role: LEADER
I20260808 02:29:55.374120 10440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:55.374042 10453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77e3533d454f9092bef2ed7a532dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260808 02:29:55.378604 10455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c77e3533d454f9092bef2ed7a532dcd. Latest consensus state: current_term: 1 leader_uuid: "4c77e3533d454f9092bef2ed7a532dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77e3533d454f9092bef2ed7a532dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } } }
I20260808 02:29:55.379298 10455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:55.378944 10454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c77e3533d454f9092bef2ed7a532dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77e3533d454f9092bef2ed7a532dcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } } }
I20260808 02:29:55.379841 10454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:55.389726 10464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:55.394546 10464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:55.403242 10464 catalog_manager.cc:1347] Generated new cluster ID: 73818a1bf74a4484a685db670a3b8c91
I20260808 02:29:55.403362 10464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:55.415748 10464 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:55.417688 10464 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:55.439868 10464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c77e3533d454f9092bef2ed7a532dcd: Generated new TSK 0
I20260808 02:29:55.440737 10464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:55.452690 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10368
W20260808 02:29:55.453938 10435 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.175:46417
W20260808 02:29:55.454861 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:29:55.454995 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10368
W20260808 02:29:55.477728 10352 connection.cc:535] client connection to 127.0.0.1:43973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43973): Connection reset by peer (error 104)
2026-08-08T02:29:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:55 dist-test-slave-pgkr krb5kdc[10475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:55 dist-test-slave-pgkr krb5kdc[10475](info): set up 2 sockets
Aug 08 02:29:55 dist-test-slave-pgkr krb5kdc[10475](info): commencing operation
krb5kdc: starting...
W20260808 02:29:56.642215 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:56 dist-test-slave-pgkr krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:57.010524 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:57.371260 10489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:57.371582 10489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:57.371680 10489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:57.378993 10489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:57.379112 10489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:57.379173 10489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:57.379238 10489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:57.379283 10489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:57.389200 10489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:57.390584 10489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:57.392045 10489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:57.405184 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:57.405156 10499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:57.405169 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:57.406167 10489 server_base.cc:1034] running on GCE node
I20260808 02:29:57.407058 10489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:57.408517 10489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:57.409786 10489 hybrid_clock.cc:648] HybridClock initialized: now 1786156197409730 us; error 36 us; skew 500 ppm
I20260808 02:29:57.410177 10489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:57 dist-test-slave-pgkr krb5kdc[10475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:57.413617 10489 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:57.415371 10489 webserver.cc:460] Webserver started at http://127.0.0.1:39883/ using document root <none> and password file <none>
I20260808 02:29:57.416088 10489 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:57.416244 10489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:57.416567 10489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:57.419801 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "7823626e20f8447785d993a6d1158c64"
format_stamp: "Formatted at 2026-08-08 02:29:57 on dist-test-slave-pgkr"
server_key: "0fd51814aab7fa275f375e0f72b4f1d7"
server_key_iv: "bc1f0e357b431a9145e0c2b1588278bd"
server_key_version: "encryptionkey@0"
I20260808 02:29:57.420675 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "7823626e20f8447785d993a6d1158c64"
format_stamp: "Formatted at 2026-08-08 02:29:57 on dist-test-slave-pgkr"
server_key: "0fd51814aab7fa275f375e0f72b4f1d7"
server_key_iv: "bc1f0e357b431a9145e0c2b1588278bd"
server_key_version: "encryptionkey@0"
I20260808 02:29:57.426246 10489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:29:57.429849 10506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:57.431248 10489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:29:57.431450 10489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "7823626e20f8447785d993a6d1158c64"
format_stamp: "Formatted at 2026-08-08 02:29:57 on dist-test-slave-pgkr"
server_key: "0fd51814aab7fa275f375e0f72b4f1d7"
server_key_iv: "bc1f0e357b431a9145e0c2b1588278bd"
server_key_version: "encryptionkey@0"
I20260808 02:29:57.431659 10489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:57.437830 10489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:57.442219 10489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:57.442528 10489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:57.462442 10489 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46713
I20260808 02:29:57.462497 10563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46713 every 8 connection(s)
I20260808 02:29:57.463999 10489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:57.466967 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10489
I20260808 02:29:57.467187 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:57.467662 20380 external_mini_cluster.cc:1428] Setting key 25ff323e809dd00d751d7425589edbfd
I20260808 02:29:57.468839 10564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:57.477955 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:57.469805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:49009 (local address 127.0.0.1:46713)
0808 02:29:57.470894 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:57.470912 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:57.470971 (+ 59us) server_negotiation.cc:407] Connection header received
0808 02:29:57.473159 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:57.473198 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:57.473788 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:57.474310 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:57.475580 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:57.475590 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:57.475612 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:57.475650 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:57.475754 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:57.475794 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:57.476071 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:57.476228 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 02:29:57.476466 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":443,"threads_started":1}
I20260808 02:29:57.482352 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:57.486348 10564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64: Bootstrap starting.
I20260808 02:29:57.490550 10564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:57.491128 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:57.488343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:41925 (local address 127.0.0.1:46713)
0808 02:29:57.488537 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:57.488543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:57.488894 (+ 351us) server_negotiation.cc:407] Connection header received
0808 02:29:57.489023 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:57.489035 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:57.489098 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:57.489215 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:57.490286 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:57.490294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:57.490303 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:29:57.490325 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:57.490376 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:29:57.490423 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:57.490560 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:57.490739 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 02:29:57.490807 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:29:57.492178 10564 log.cc:824] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:57.495332 10564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64: No bootstrap required, opened a new log
I20260808 02:29:57.503257 10564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7823626e20f8447785d993a6d1158c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260808 02:29:57.503638 10564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:57.503715 10564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7823626e20f8447785d993a6d1158c64, State: Initialized, Role: FOLLOWER
I20260808 02:29:57.504235 10564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7823626e20f8447785d993a6d1158c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260808 02:29:57.504431 10564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:57.504541 10564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:57.504680 10564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:57.507004 10564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7823626e20f8447785d993a6d1158c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260808 02:29:57.507542 10564 leader_election.cc:302] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7823626e20f8447785d993a6d1158c64; no voters:
I20260808 02:29:57.508142 10564 leader_election.cc:288] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:57.508301 10575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:57.509042 10575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [term 1 LEADER]: Becoming Leader. State: Replica: 7823626e20f8447785d993a6d1158c64, State: Running, Role: LEADER
I20260808 02:29:57.509573 10575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7823626e20f8447785d993a6d1158c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } }
I20260808 02:29:57.510311 10564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:57.512631 10577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7823626e20f8447785d993a6d1158c64. Latest consensus state: current_term: 1 leader_uuid: "7823626e20f8447785d993a6d1158c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7823626e20f8447785d993a6d1158c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } } }
I20260808 02:29:57.513118 10577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:57.513556 10576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7823626e20f8447785d993a6d1158c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7823626e20f8447785d993a6d1158c64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46713 } } }
I20260808 02:29:57.513988 10576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64 [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:57.515631 10581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:57.520870 10581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:57.530576 10581 catalog_manager.cc:1347] Generated new cluster ID: ed8020aa281b4303863d9b513be541ef
I20260808 02:29:57.530755 10581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:57.545125 10581 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:57.545629 10581 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:57.552868 10581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7823626e20f8447785d993a6d1158c64: Generated new TSK 0
I20260808 02:29:57.553681 10581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:57.559382 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10489
W20260808 02:29:57.560614 10559 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.175:49009
W20260808 02:29:57.561386 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10489: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:29:57.561512 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10489
2026-08-08T02:29:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2106 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:57 dist-test-slave-pgkr krb5kdc[10599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:57 dist-test-slave-pgkr krb5kdc[10599](info): set up 2 sockets
Aug 08 02:29:57 dist-test-slave-pgkr krb5kdc[10599](info): commencing operation
krb5kdc: starting...
W20260808 02:29:58.774799 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:29:58 dist-test-slave-pgkr krb5kdc[10599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:29:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:29:59.134131 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:29:59.499646 10613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:29:59.499955 10613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:29:59.500145 10613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:29:59.507551 10613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:29:59.507671 10613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:29:59.507787 10613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:29:59.507920 10613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:29:59.508000 10613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:29:59.518205 10613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:29:59.519646 10613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:29:59.521090 10613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:29:59.534015 10619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:59.534010 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:29:59.534588 10622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:29:59.535290 10613 server_base.cc:1034] running on GCE node
I20260808 02:29:59.536113 10613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:29:59.537647 10613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:29:59.538888 10613 hybrid_clock.cc:648] HybridClock initialized: now 1786156199538838 us; error 41 us; skew 500 ppm
I20260808 02:29:59.539292 10613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:29:59 dist-test-slave-pgkr krb5kdc[10599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:29:59.542826 10613 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:29:59.544375 10613 webserver.cc:460] Webserver started at http://127.0.0.1:45563/ using document root <none> and password file <none>
I20260808 02:29:59.545082 10613 fs_manager.cc:359] Metadata directory not provided
I20260808 02:29:59.545241 10613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:29:59.545565 10613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:29:59.548684 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "30d45f26b3df4bc2a370c441e5467bee"
format_stamp: "Formatted at 2026-08-08 02:29:59 on dist-test-slave-pgkr"
server_key: "9614acfce0d79170f30b468527304691"
server_key_iv: "bf0ca1f7b4b2dec6cd41d40531d4b150"
server_key_version: "encryptionkey@0"
I20260808 02:29:59.549467 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "30d45f26b3df4bc2a370c441e5467bee"
format_stamp: "Formatted at 2026-08-08 02:29:59 on dist-test-slave-pgkr"
server_key: "9614acfce0d79170f30b468527304691"
server_key_iv: "bf0ca1f7b4b2dec6cd41d40531d4b150"
server_key_version: "encryptionkey@0"
I20260808 02:29:59.554749 10613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:29:59.558296 10629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:29:59.559784 10613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:29:59.560011 10613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "30d45f26b3df4bc2a370c441e5467bee"
format_stamp: "Formatted at 2026-08-08 02:29:59 on dist-test-slave-pgkr"
server_key: "9614acfce0d79170f30b468527304691"
server_key_iv: "bf0ca1f7b4b2dec6cd41d40531d4b150"
server_key_version: "encryptionkey@0"
I20260808 02:29:59.560222 10613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:29:59.577915 10613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:29:59.582068 10613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:29:59.582309 10613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:29:59.600893 10613 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35203
I20260808 02:29:59.600924 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35203 every 8 connection(s)
I20260808 02:29:59.602286 10613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:29:59.606097 10686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:29:59.610720 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10613
I20260808 02:29:59.610926 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:29:59.611553 20380 external_mini_cluster.cc:1428] Setting key bc3e86d6cafdbb5ad9216caf0d1a6cbb
I20260808 02:29:59.617928 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee: Bootstrap starting.
I20260808 02:29:59.621692 10686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee: Neither blocks nor log segments found. Creating new log.
I20260808 02:29:59.622995 10686 log.cc:824] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee: Log is configured to *not* fsync() on all Append() calls
I20260808 02:29:59.625993 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee: No bootstrap required, opened a new log
Aug 08 02:29:59 dist-test-slave-pgkr krb5kdc[10599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 02:29:59.635113 10686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d45f26b3df4bc2a370c441e5467bee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260808 02:29:59.635510 10686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:29:59.635648 10686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d45f26b3df4bc2a370c441e5467bee, State: Initialized, Role: FOLLOWER
I20260808 02:29:59.636055 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:59.613462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:52405 (local address 127.0.0.1:35203)
0808 02:29:59.614290 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:59.614301 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:59.614350 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:29:59.616390 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:59.616413 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:59.616798 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:59.617122 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:59.619092 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:59.620411 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:59.622322 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:59.623311 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:59.623492 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:59.628112 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:59.628123 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:59.628137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:59.628174 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:59.631494 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:59.632320 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:59.632327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:59.632333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:59.632503 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:59.633004 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:59.633008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:59.633010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:59.633562 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:59.633935 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:59.634474 (+ 539us) server_negotiation.cc:299] Negotiation successful
0808 02:29:59.634828 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":385,"threads_started":1}
I20260808 02:29:59.636348 10686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d45f26b3df4bc2a370c441e5467bee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260808 02:29:59.636641 10686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:29:59.636817 10686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:29:59.637017 10686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:29:59.639899 10686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d45f26b3df4bc2a370c441e5467bee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260808 02:29:59.640085 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:29:59.640554 10686 leader_election.cc:302] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d45f26b3df4bc2a370c441e5467bee; no voters:
I20260808 02:29:59.641191 10686 leader_election.cc:288] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:29:59.641350 10692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:29:59.642912 10692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [term 1 LEADER]: Becoming Leader. State: Replica: 30d45f26b3df4bc2a370c441e5467bee, State: Running, Role: LEADER
I20260808 02:29:59.643461 10686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:29:59.643567 10692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d45f26b3df4bc2a370c441e5467bee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260808 02:29:59.650395 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d45f26b3df4bc2a370c441e5467bee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d45f26b3df4bc2a370c441e5467bee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } } }
I20260808 02:29:59.650883 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:59.651508 10695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d45f26b3df4bc2a370c441e5467bee. Latest consensus state: current_term: 1 leader_uuid: "30d45f26b3df4bc2a370c441e5467bee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d45f26b3df4bc2a370c441e5467bee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } } }
I20260808 02:29:59.651852 10695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee [sys.catalog]: This master's current role is: LEADER
I20260808 02:29:59.656850 10708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:29:59.664182 10708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:29:59.681870 10708 catalog_manager.cc:1347] Generated new cluster ID: 80be1b7c6a354676a87aec5d8d324f81
I20260808 02:29:59.681993 10708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:29:59.683416 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:29:59.649832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:51071 (local address 127.0.0.1:35203)
0808 02:29:59.655304 (+ 5472us) server_negotiation.cc:206] Beginning negotiation
0808 02:29:59.655317 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:29:59.660973 (+ 5656us) server_negotiation.cc:407] Connection header received
0808 02:29:59.661345 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:29:59.661351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:29:59.661431 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:29:59.661553 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:29:59.663519 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:59.665383 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:59.667984 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:29:59.668979 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:29:59.669115 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:29:59.673584 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:29:59.673595 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:29:59.673603 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:29:59.673654 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:29:59.678530 (+ 4876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:59.679839 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:59.679847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:59.679854 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:59.680047 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:29:59.680805 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:29:59.680813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:29:59.680819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:29:59.681357 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:29:59.681688 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 02:29:59.682563 (+ 875us) server_negotiation.cc:299] Negotiation successful
0808 02:29:59.682850 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5270}
I20260808 02:29:59.703356 10708 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:29:59.704653 10708 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:29:59.715162 10708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d45f26b3df4bc2a370c441e5467bee: Generated new TSK 0
I20260808 02:29:59.715996 10708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:29:59.728412 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10613
I20260808 02:29:59.729895 10680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:29:59.822355 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10613
W20260808 02:29:59.842968 10597 connection.cc:535] client connection to 127.0.0.1:35203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35203): Connection reset by peer (error 104)
2026-08-08T02:29:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:29:59 dist-test-slave-pgkr krb5kdc[10721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:29:59 dist-test-slave-pgkr krb5kdc[10721](info): set up 2 sockets
Aug 08 02:29:59 dist-test-slave-pgkr krb5kdc[10721](info): commencing operation
krb5kdc: starting...
W20260808 02:30:01.016081 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.003s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:01 dist-test-slave-pgkr krb5kdc[10721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:30:01.392225 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:01.768653 10735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:01.768961 10735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:01.769135 10735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:01.776732 10735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:01.776844 10735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:01.776958 10735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:01.777087 10735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:01.777156 10735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:01.787313 10735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:01.788683 10735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:01.790041 10735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:01.802338 10740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:01.802287 10741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:01.802377 10743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:01.803370 10735 server_base.cc:1034] running on GCE node
I20260808 02:30:01.804286 10735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:01.805802 10735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:01.807018 10735 hybrid_clock.cc:648] HybridClock initialized: now 1786156201806960 us; error 47 us; skew 500 ppm
I20260808 02:30:01.807382 10735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:01 dist-test-slave-pgkr krb5kdc[10721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:01.810750 10735 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:30:01.812332 10735 webserver.cc:460] Webserver started at http://127.0.0.1:39885/ using document root <none> and password file <none>
I20260808 02:30:01.813040 10735 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:01.813180 10735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:01.813498 10735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:01.816643 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f"
format_stamp: "Formatted at 2026-08-08 02:30:01 on dist-test-slave-pgkr"
server_key: "cc4831c9284003253b99f3870d69867e"
server_key_iv: "918af9fe3204c740343451e6edd0584f"
server_key_version: "encryptionkey@0"
I20260808 02:30:01.817441 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f"
format_stamp: "Formatted at 2026-08-08 02:30:01 on dist-test-slave-pgkr"
server_key: "cc4831c9284003253b99f3870d69867e"
server_key_iv: "918af9fe3204c740343451e6edd0584f"
server_key_version: "encryptionkey@0"
I20260808 02:30:01.822831 10735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:30:01.826457 10752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:01.827970 10735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:30:01.828166 10735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f"
format_stamp: "Formatted at 2026-08-08 02:30:01 on dist-test-slave-pgkr"
server_key: "cc4831c9284003253b99f3870d69867e"
server_key_iv: "918af9fe3204c740343451e6edd0584f"
server_key_version: "encryptionkey@0"
I20260808 02:30:01.828377 10735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:01.852843 10735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:01.857261 10735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:01.857580 10735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:01.877913 10735 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45091
I20260808 02:30:01.878005 10804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45091 every 8 connection(s)
I20260808 02:30:01.879391 10735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:01.883469 10805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:01.885701 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10735
I20260808 02:30:01.885893 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:01.886441 20380 external_mini_cluster.cc:1428] Setting key e6621be3026a290f11b3d9ad2743ac54
I20260808 02:30:01.896517 10805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f: Bootstrap starting.
I20260808 02:30:01.899920 10805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:01.900718 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:01.888386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:32817 (local address 127.0.0.1:45091)
0808 02:30:01.889241 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:01.889256 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:01.889303 (+ 47us) server_negotiation.cc:407] Connection header received
0808 02:30:01.891116 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:01.891140 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:01.891545 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:01.891890 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:01.892580 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:01.894028 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:01.895947 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:01.896665 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:01.896877 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:01.898184 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:01.898192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:01.898207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:01.898848 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:01.898930 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:01.898955 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:01.899122 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:01.899426 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 02:30:01.899602 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":404,"threads_started":1}
I20260808 02:30:01.901176 10805 log.cc:824] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:01.904330 10805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f: No bootstrap required, opened a new log
I20260808 02:30:01.904737 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:01.913252 10805 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260808 02:30:01.913774 10805 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:01.913900 10805 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bbb35174ec4cf1a2db2c2f2f93c49f, State: Initialized, Role: FOLLOWER
I20260808 02:30:01.914695 10805 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260808 02:30:01.914976 10805 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:01.915149 10805 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:01.915378 10805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:01.916602 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:01.909325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:33353 (local address 127.0.0.1:45091)
0808 02:30:01.909524 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:01.909530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:01.910027 (+ 497us) server_negotiation.cc:407] Connection header received
0808 02:30:01.910189 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:01.910195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:01.910276 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:01.910382 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:01.910992 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:01.912193 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:01.913503 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:01.914177 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:01.914304 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:01.915757 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:01.915763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:01.915771 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:01.915793 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:01.915869 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:01.915890 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:01.916017 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:01.916167 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:30:01.916232 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:30:01.918702 10805 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260808 02:30:01.919245 10805 leader_election.cc:302] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9bbb35174ec4cf1a2db2c2f2f93c49f; no voters:
I20260808 02:30:01.919848 10805 leader_election.cc:288] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:01.920020 10816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:01.920849 10816 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [term 1 LEADER]: Becoming Leader. State: Replica: f9bbb35174ec4cf1a2db2c2f2f93c49f, State: Running, Role: LEADER
I20260808 02:30:01.921504 10816 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } }
I20260808 02:30:01.922214 10805 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:01.925140 10817 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } } }
I20260808 02:30:01.926038 10817 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:01.925169 10818 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9bbb35174ec4cf1a2db2c2f2f93c49f. Latest consensus state: current_term: 1 leader_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bbb35174ec4cf1a2db2c2f2f93c49f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45091 } } }
I20260808 02:30:01.926529 10818 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:01.928122 10827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:01.933274 10827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:01.941931 10827 catalog_manager.cc:1347] Generated new cluster ID: fd9b12d3e7254c44b7959a15257ff110
I20260808 02:30:01.942067 10827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:01.973255 10827 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:01.975243 10827 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:01.989223 10827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9bbb35174ec4cf1a2db2c2f2f93c49f: Generated new TSK 0
I20260808 02:30:01.990429 10827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:02.000463 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10735
W20260808 02:30:02.002146 10800 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.175:32817
W20260808 02:30:02.003005 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10735: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:30:02.003172 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10735
W20260808 02:30:02.026571 10719 connection.cc:535] client connection to 127.0.0.1:45091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45091): Connection reset by peer (error 104)
2026-08-08T02:30:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:02 dist-test-slave-pgkr krb5kdc[10840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:02 dist-test-slave-pgkr krb5kdc[10840](info): set up 2 sockets
Aug 08 02:30:02 dist-test-slave-pgkr krb5kdc[10840](info): commencing operation
krb5kdc: starting...
W20260808 02:30:03.191172 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:03 dist-test-slave-pgkr krb5kdc[10840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 02:30:03.544214 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:03.912425 10853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:03.912802 10853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:03.912995 10853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:03.920543 10853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:03.920713 10853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:03.920833 10853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:03.920958 10853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:03.921053 10853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:03.931309 10853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:03.932814 10853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:03.934275 10853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:03.947649 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:03.947773 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:03.948632 10862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:03.949054 10853 server_base.cc:1034] running on GCE node
I20260808 02:30:03.949965 10853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:03.951507 10853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:03.952759 10853 hybrid_clock.cc:648] HybridClock initialized: now 1786156203952710 us; error 58 us; skew 500 ppm
I20260808 02:30:03.953137 10853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:03 dist-test-slave-pgkr krb5kdc[10840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:03.961372 10853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 02:30:03.963088 10853 webserver.cc:460] Webserver started at http://127.0.0.1:40795/ using document root <none> and password file <none>
I20260808 02:30:03.963908 10853 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:03.964067 10853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:03.964392 10853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:03.968726 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "3f8adf61090c423792e7bba34fb3cf34"
format_stamp: "Formatted at 2026-08-08 02:30:03 on dist-test-slave-pgkr"
server_key: "d1a81fc6672a1d2b0a03d92e43b38154"
server_key_iv: "76cf25d72eb40ebc6e4d7de1250c5333"
server_key_version: "encryptionkey@0"
I20260808 02:30:03.969605 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "3f8adf61090c423792e7bba34fb3cf34"
format_stamp: "Formatted at 2026-08-08 02:30:03 on dist-test-slave-pgkr"
server_key: "d1a81fc6672a1d2b0a03d92e43b38154"
server_key_iv: "76cf25d72eb40ebc6e4d7de1250c5333"
server_key_version: "encryptionkey@0"
I20260808 02:30:03.975241 10853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 02:30:03.978962 10870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:03.980376 10853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:30:03.980573 10853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "3f8adf61090c423792e7bba34fb3cf34"
format_stamp: "Formatted at 2026-08-08 02:30:03 on dist-test-slave-pgkr"
server_key: "d1a81fc6672a1d2b0a03d92e43b38154"
server_key_iv: "76cf25d72eb40ebc6e4d7de1250c5333"
server_key_version: "encryptionkey@0"
I20260808 02:30:03.980798 10853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:03.987027 10853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:03.991375 10853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:03.991696 10853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:04.011240 10853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43031
I20260808 02:30:04.011327 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43031 every 8 connection(s)
I20260808 02:30:04.012720 10853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:04.016602 10927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:04.019930 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10853
I20260808 02:30:04.020130 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:04.020681 20380 external_mini_cluster.cc:1428] Setting key fb8235ec4d0037012029f3046999ab7e
I20260808 02:30:04.028829 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:04.022750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:45423 (local address 127.0.0.1:43031)
0808 02:30:04.023521 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:04.023533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:04.023576 (+ 43us) server_negotiation.cc:407] Connection header received
0808 02:30:04.025301 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:04.025322 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:04.025681 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:04.026089 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:04.027228 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:04.027234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:04.027245 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:04.027258 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:04.027315 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:04.027333 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:04.027576 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:04.027731 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 02:30:04.027914 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":343,"threads_started":1}
I20260808 02:30:04.029064 10927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34: Bootstrap starting.
I20260808 02:30:04.032600 10927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:04.032655 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:04.033677 10927 log.cc:824] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:04.036685 10927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34: No bootstrap required, opened a new log
I20260808 02:30:04.039986 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:04.037379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:54937 (local address 127.0.0.1:43031)
0808 02:30:04.037598 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:04.037603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:04.037947 (+ 344us) server_negotiation.cc:407] Connection header received
0808 02:30:04.038143 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:04.038147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:04.038199 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:04.038281 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:04.039294 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:04.039298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:04.039301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:04.039316 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:04.039358 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:04.039372 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:04.039464 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:04.039603 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 02:30:04.039680 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:30:04.045569 10927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8adf61090c423792e7bba34fb3cf34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260808 02:30:04.046073 10927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:04.046222 10927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8adf61090c423792e7bba34fb3cf34, State: Initialized, Role: FOLLOWER
I20260808 02:30:04.047012 10927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8adf61090c423792e7bba34fb3cf34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260808 02:30:04.047281 10927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:04.047461 10927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:04.047636 10927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:04.049908 10927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8adf61090c423792e7bba34fb3cf34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260808 02:30:04.050458 10927 leader_election.cc:302] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f8adf61090c423792e7bba34fb3cf34; no voters:
I20260808 02:30:04.051165 10927 leader_election.cc:288] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:04.051323 10937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:04.052429 10937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [term 1 LEADER]: Becoming Leader. State: Replica: 3f8adf61090c423792e7bba34fb3cf34, State: Running, Role: LEADER
I20260808 02:30:04.053124 10937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8adf61090c423792e7bba34fb3cf34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260808 02:30:04.053884 10927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:04.056413 10938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f8adf61090c423792e7bba34fb3cf34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8adf61090c423792e7bba34fb3cf34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } } }
I20260808 02:30:04.056866 10938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:04.058483 10939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8adf61090c423792e7bba34fb3cf34. Latest consensus state: current_term: 1 leader_uuid: "3f8adf61090c423792e7bba34fb3cf34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8adf61090c423792e7bba34fb3cf34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } } }
I20260808 02:30:04.058959 10939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:04.063578 10945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:04.069721 10945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:04.079701 10945 catalog_manager.cc:1347] Generated new cluster ID: 4f833a4d956549f5914a41b063e50509
I20260808 02:30:04.079838 10945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:04.094179 10945 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:04.094678 10945 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:04.101742 10945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f8adf61090c423792e7bba34fb3cf34: Generated new TSK 0
I20260808 02:30:04.102550 10945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:04.115028 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10853
W20260808 02:30:04.116886 10922 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.175:45423
W20260808 02:30:04.117653 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10853: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:30:04.117811 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10853
2026-08-08T02:30:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2112 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:04 dist-test-slave-pgkr krb5kdc[10961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:04 dist-test-slave-pgkr krb5kdc[10961](info): set up 2 sockets
Aug 08 02:30:04 dist-test-slave-pgkr krb5kdc[10961](info): commencing operation
krb5kdc: starting...
W20260808 02:30:05.316347 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:05 dist-test-slave-pgkr krb5kdc[10961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:05.677947 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37709
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:06.036779 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:06.037076 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:06.037173 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:06.044404 10976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:06.044517 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:06.044564 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:06.044634 10976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:06.044679 10976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:06.054455 10976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:06.055846 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:06.057171 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:06.069537 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:06.069587 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:06.069926 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:06.070711 10976 server_base.cc:1034] running on GCE node
I20260808 02:30:06.071509 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:06.073004 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:06.074267 10976 hybrid_clock.cc:648] HybridClock initialized: now 1786156206074225 us; error 60 us; skew 500 ppm
I20260808 02:30:06.074718 10976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:06 dist-test-slave-pgkr krb5kdc[10961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156206, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:06.078112 10976 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:06.079706 10976 webserver.cc:460] Webserver started at http://127.19.231.62:45939/ using document root <none> and password file <none>
I20260808 02:30:06.080461 10976 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:06.080627 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:06.080957 10976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:06.084215 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "9437a4cf2c224aaaa8401865f8e7576d"
format_stamp: "Formatted at 2026-08-08 02:30:06 on dist-test-slave-pgkr"
server_key: "a2754788064d7d3b6582540f2031fb10"
server_key_iv: "bfd9b0558ee0b4116b3128c94489a80d"
server_key_version: "encryptionkey@0"
I20260808 02:30:06.084995 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "9437a4cf2c224aaaa8401865f8e7576d"
format_stamp: "Formatted at 2026-08-08 02:30:06 on dist-test-slave-pgkr"
server_key: "a2754788064d7d3b6582540f2031fb10"
server_key_iv: "bfd9b0558ee0b4116b3128c94489a80d"
server_key_version: "encryptionkey@0"
I20260808 02:30:06.090394 10976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 02:30:06.094309 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:06.095813 10976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:30:06.096012 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "9437a4cf2c224aaaa8401865f8e7576d"
format_stamp: "Formatted at 2026-08-08 02:30:06 on dist-test-slave-pgkr"
server_key: "a2754788064d7d3b6582540f2031fb10"
server_key_iv: "bfd9b0558ee0b4116b3128c94489a80d"
server_key_version: "encryptionkey@0"
I20260808 02:30:06.096242 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:06.109951 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:06.114244 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:06.114543 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:06.134703 10976 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:37709
I20260808 02:30:06.134807 11044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:37709 every 8 connection(s)
I20260808 02:30:06.136121 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:06.140316 11045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:06.143441 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 10976
I20260808 02:30:06.143659 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:06.144289 20380 external_mini_cluster.cc:1428] Setting key 885f6da22c6757114fa87e250a1bd13a
I20260808 02:30:06.154241 11045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d: Bootstrap starting.
I20260808 02:30:06.158708 11045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:06.160194 11045 log.cc:824] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d: Log is configured to *not* fsync() on all Append() calls
Aug 08 02:30:06 dist-test-slave-pgkr krb5kdc[10961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:06.163743 11045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d: No bootstrap required, opened a new log
I20260808 02:30:06.169975 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:06.146270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38385 (local address 127.19.231.62:37709)
0808 02:30:06.147162 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:06.147173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:06.147219 (+ 46us) server_negotiation.cc:407] Connection header received
0808 02:30:06.148897 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:06.148920 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:06.149330 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:06.149647 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:06.151677 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:06.153040 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:06.154976 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:06.155679 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:06.155828 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:06.161808 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:06.161816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:06.161830 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:06.161869 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:06.165146 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:06.166245 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:06.166254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:06.166260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:06.166436 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:06.167129 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:06.167133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:06.167136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:06.167695 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:06.167966 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:06.168518 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 02:30:06.168834 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":437,"threads_started":1}
I20260808 02:30:06.174803 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:06.174461 11045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9437a4cf2c224aaaa8401865f8e7576d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37709 } }
I20260808 02:30:06.175020 11045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:06.175130 11045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9437a4cf2c224aaaa8401865f8e7576d, State: Initialized, Role: FOLLOWER
I20260808 02:30:06.175796 11045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9437a4cf2c224aaaa8401865f8e7576d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37709 } }
I20260808 02:30:06.176115 11045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:06.176266 11045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:06.176440 11045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:06.180048 11045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9437a4cf2c224aaaa8401865f8e7576d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37709 } }
I20260808 02:30:06.180778 11045 leader_election.cc:302] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9437a4cf2c224aaaa8401865f8e7576d; no voters:
I20260808 02:30:06.181519 11045 leader_election.cc:288] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:06.181656 11056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:06.184149 11045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:06.184091 11056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [term 1 LEADER]: Becoming Leader. State: Replica: 9437a4cf2c224aaaa8401865f8e7576d, State: Running, Role: LEADER
I20260808 02:30:06.184737 11056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9437a4cf2c224aaaa8401865f8e7576d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37709 } }
I20260808 02:30:06.188441 11057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9437a4cf2c224aaaa8401865f8e7576d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9437a4cf2c224aaaa8401865f8e7576d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37709 } } }
I20260808 02:30:06.188947 11057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:06.188421 11058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9437a4cf2c224aaaa8401865f8e7576d. Latest consensus state: current_term: 1 leader_uuid: "9437a4cf2c224aaaa8401865f8e7576d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9437a4cf2c224aaaa8401865f8e7576d" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37709 } } }
I20260808 02:30:06.189443 11058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:06.196601 11067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:06.202033 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:06.180428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52683 (local address 127.19.231.62:37709)
0808 02:30:06.180667 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:06.180674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:06.181086 (+ 412us) server_negotiation.cc:407] Connection header received
0808 02:30:06.181261 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:06.181270 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:06.181389 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:06.181553 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:06.183787 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:06.185115 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:06.187247 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:06.188114 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:06.188293 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:06.192966 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:06.192977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:06.192986 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:06.193041 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:06.197474 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:06.198827 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:06.198836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:06.198841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:06.199049 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:06.199750 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:06.199757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:06.199760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:06.200287 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:06.200614 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:06.201410 (+ 796us) server_negotiation.cc:299] Negotiation successful
0808 02:30:06.201630 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":107}
I20260808 02:30:06.203151 11067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:06.213199 11067 catalog_manager.cc:1347] Generated new cluster ID: a92a4dc73a5d4358842bb1762989f0fd
I20260808 02:30:06.213325 11067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:06.250248 11067 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:06.251791 11067 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:06.259202 11067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9437a4cf2c224aaaa8401865f8e7576d: Generated new TSK 0
I20260808 02:30:06.260275 11067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:06.267246 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 10976
I20260808 02:30:06.268769 11040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:06.362713 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 10976
W20260808 02:30:06.384866 10959 connection.cc:535] client connection to 127.19.231.62:37709 recv error: Network error: failed to read from TLS socket (remote: 127.19.231.62:37709): Connection reset by peer (error 104)
2026-08-08T02:30:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:06 dist-test-slave-pgkr krb5kdc[11080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:06 dist-test-slave-pgkr krb5kdc[11080](info): set up 2 sockets
Aug 08 02:30:06 dist-test-slave-pgkr krb5kdc[11080](info): commencing operation
krb5kdc: starting...
W20260808 02:30:07.575475 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.012s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:07 dist-test-slave-pgkr krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:07.930456 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41833
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:41833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:08.291050 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:08.291354 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:08.291543 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:08.299132 11095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:08.299256 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:08.299336 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:08.299453 11095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:08.299530 11095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:08.310021 11095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:41833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:41833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:08.311448 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:08.312862 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:08.325323 11101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:08.325372 11102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:08.325420 11104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:08.326292 11095 server_base.cc:1034] running on GCE node
I20260808 02:30:08.327144 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:08.328550 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:08.329792 11095 hybrid_clock.cc:648] HybridClock initialized: now 1786156208329750 us; error 37 us; skew 500 ppm
I20260808 02:30:08.330153 11095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:08 dist-test-slave-pgkr krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156208, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:08.333472 11095 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:08.335070 11095 webserver.cc:460] Webserver started at http://127.19.231.62:34863/ using document root <none> and password file <none>
I20260808 02:30:08.335776 11095 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:08.335929 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:08.336247 11095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:08.339334 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "02c58474014b42b6a534431ee4690f34"
format_stamp: "Formatted at 2026-08-08 02:30:08 on dist-test-slave-pgkr"
server_key: "eab0de42335e590782bc0bab3e39a7e4"
server_key_iv: "16d85333071c0d9fa3ad36d56f0a72e2"
server_key_version: "encryptionkey@0"
I20260808 02:30:08.340128 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "02c58474014b42b6a534431ee4690f34"
format_stamp: "Formatted at 2026-08-08 02:30:08 on dist-test-slave-pgkr"
server_key: "eab0de42335e590782bc0bab3e39a7e4"
server_key_iv: "16d85333071c0d9fa3ad36d56f0a72e2"
server_key_version: "encryptionkey@0"
I20260808 02:30:08.345644 11095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:30:08.349480 11112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:08.350946 11095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:30:08.351150 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "02c58474014b42b6a534431ee4690f34"
format_stamp: "Formatted at 2026-08-08 02:30:08 on dist-test-slave-pgkr"
server_key: "eab0de42335e590782bc0bab3e39a7e4"
server_key_iv: "16d85333071c0d9fa3ad36d56f0a72e2"
server_key_version: "encryptionkey@0"
I20260808 02:30:08.351363 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:08.378250 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:08.382531 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:08.382870 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:08.403309 11095 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:41833
I20260808 02:30:08.403319 11166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:41833 every 8 connection(s)
I20260808 02:30:08.404822 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:08.406381 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11095
I20260808 02:30:08.406577 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:08.407198 20380 external_mini_cluster.cc:1428] Setting key c09af4681974732da896218114138dce
I20260808 02:30:08.409721 11167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:08.420920 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:08.409086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55911 (local address 127.19.231.62:41833)
0808 02:30:08.409958 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:08.409971 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:08.410016 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:30:08.411756 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:08.411779 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:08.412197 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:08.412548 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:08.413171 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:08.414495 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:08.416414 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:08.417068 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:08.417234 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:08.418496 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:08.418502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:08.418514 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:08.419174 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:08.419251 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:08.419279 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:08.419403 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:08.419762 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 02:30:08.419966 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":377,"threads_started":1}
I20260808 02:30:08.422422 11167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34: Bootstrap starting.
I20260808 02:30:08.424976 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:08.425917 11167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:08.427014 11167 log.cc:824] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:08.430225 11167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34: No bootstrap required, opened a new log
I20260808 02:30:08.436682 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:08.429653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54373 (local address 127.19.231.62:41833)
0808 02:30:08.429870 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:08.429875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:08.430176 (+ 301us) server_negotiation.cc:407] Connection header received
0808 02:30:08.430361 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:08.430368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:08.430438 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:08.430519 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:08.431223 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:08.431993 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:08.433993 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:08.434459 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:08.434557 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:08.435831 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:08.435836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:08.435842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:08.435859 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:08.435917 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:08.435932 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:08.436017 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:08.436253 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 02:30:08.436333 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:30:08.440115 11167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c58474014b42b6a534431ee4690f34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41833 } }
I20260808 02:30:08.440511 11167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:08.440655 11167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c58474014b42b6a534431ee4690f34, State: Initialized, Role: FOLLOWER
I20260808 02:30:08.441300 11167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c58474014b42b6a534431ee4690f34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41833 } }
I20260808 02:30:08.441533 11167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:08.441707 11167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:08.441897 11167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:08.444257 11167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c58474014b42b6a534431ee4690f34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41833 } }
I20260808 02:30:08.444778 11167 leader_election.cc:302] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02c58474014b42b6a534431ee4690f34; no voters:
I20260808 02:30:08.445421 11167 leader_election.cc:288] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:08.445546 11177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:08.446381 11177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [term 1 LEADER]: Becoming Leader. State: Replica: 02c58474014b42b6a534431ee4690f34, State: Running, Role: LEADER
I20260808 02:30:08.447074 11177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c58474014b42b6a534431ee4690f34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41833 } }
I20260808 02:30:08.447670 11167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:08.450176 11179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02c58474014b42b6a534431ee4690f34. Latest consensus state: current_term: 1 leader_uuid: "02c58474014b42b6a534431ee4690f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c58474014b42b6a534431ee4690f34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41833 } } }
I20260808 02:30:08.450574 11179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:08.450279 11178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02c58474014b42b6a534431ee4690f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c58474014b42b6a534431ee4690f34" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 41833 } } }
I20260808 02:30:08.450758 11178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:08.451614 11184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:08.458485 11184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:08.469165 11184 catalog_manager.cc:1347] Generated new cluster ID: 1c4eef222c6b404eab7532abf8958da1
I20260808 02:30:08.469301 11184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:08.479064 11184 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:08.480490 11184 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:08.492275 11184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02c58474014b42b6a534431ee4690f34: Generated new TSK 0
I20260808 02:30:08.493425 11184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:08.502347 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11095
W20260808 02:30:08.503752 11162 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55911
W20260808 02:30:08.504603 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11095: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:30:08.504781 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11095
W20260808 02:30:08.525456 11078 connection.cc:535] client connection to 127.19.231.62:41833 recv error: Network error: failed to read from TLS socket (remote: 127.19.231.62:41833): Connection reset by peer (error 104)
2026-08-08T02:30:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2140 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:08 dist-test-slave-pgkr krb5kdc[11201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:08 dist-test-slave-pgkr krb5kdc[11201](info): set up 2 sockets
Aug 08 02:30:08 dist-test-slave-pgkr krb5kdc[11201](info): commencing operation
krb5kdc: starting...
W20260808 02:30:09.703810 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:09 dist-test-slave-pgkr krb5kdc[11201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:10.064410 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44495
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:10.422189 11215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:10.422487 11215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:10.422712 11215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:10.430068 11215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:10.430178 11215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:10.430303 11215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:10.430399 11215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:10.430482 11215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:10.440855 11215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:44495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:10.442253 11215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:10.443717 11215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:10.456365 11220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:10.456269 11225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:10.456318 11221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:10.457077 11215 server_base.cc:1034] running on GCE node
I20260808 02:30:10.457870 11215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:10.459391 11215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:10.460634 11215 hybrid_clock.cc:648] HybridClock initialized: now 1786156210460573 us; error 51 us; skew 500 ppm
I20260808 02:30:10.461055 11215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:10 dist-test-slave-pgkr krb5kdc[11201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156210, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:10.469576 11215 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:10.471441 11215 webserver.cc:460] Webserver started at http://127.19.231.62:34399/ using document root <none> and password file <none>
I20260808 02:30:10.472186 11215 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:10.472355 11215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:10.472684 11215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:10.475893 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966"
format_stamp: "Formatted at 2026-08-08 02:30:10 on dist-test-slave-pgkr"
server_key: "337854f17d63714f2aababe26ab7d56f"
server_key_iv: "1ac73f543c1330991ae3fa4c813ce6ca"
server_key_version: "encryptionkey@0"
I20260808 02:30:10.476739 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966"
format_stamp: "Formatted at 2026-08-08 02:30:10 on dist-test-slave-pgkr"
server_key: "337854f17d63714f2aababe26ab7d56f"
server_key_iv: "1ac73f543c1330991ae3fa4c813ce6ca"
server_key_version: "encryptionkey@0"
I20260808 02:30:10.482595 11215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:30:10.486292 11232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:10.487872 11215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 02:30:10.488132 11215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966"
format_stamp: "Formatted at 2026-08-08 02:30:10 on dist-test-slave-pgkr"
server_key: "337854f17d63714f2aababe26ab7d56f"
server_key_iv: "1ac73f543c1330991ae3fa4c813ce6ca"
server_key_version: "encryptionkey@0"
I20260808 02:30:10.488376 11215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:10.495128 11215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:10.499697 11215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:10.500034 11215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:10.521636 11215 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:44495
I20260808 02:30:10.521660 11289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:44495 every 8 connection(s)
I20260808 02:30:10.523267 11215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:10.527555 11290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:10.531800 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11215
I20260808 02:30:10.532004 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:10.532519 20380 external_mini_cluster.cc:1428] Setting key 19527edb57495b65008181c8409dff45
I20260808 02:30:10.539038 11290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966: Bootstrap starting.
I20260808 02:30:10.541324 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:10.534451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49647 (local address 127.19.231.62:44495)
0808 02:30:10.535448 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:10.535461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:10.535506 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:30:10.537613 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:10.537638 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:10.537993 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:10.538340 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:10.539608 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:10.539615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:10.539629 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:10.539647 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:10.539708 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:10.539729 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:10.539907 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:10.540081 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 02:30:10.540311 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":485,"threads_started":1}
I20260808 02:30:10.542870 11290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:10.544071 11290 log.cc:824] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:10.545768 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:10.547618 11290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966: No bootstrap required, opened a new log
I20260808 02:30:10.553750 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:10.551211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46089 (local address 127.19.231.62:44495)
0808 02:30:10.551403 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:10.551413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:10.551818 (+ 405us) server_negotiation.cc:407] Connection header received
0808 02:30:10.551956 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:10.551965 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:10.552028 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:10.552140 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:10.553067 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:10.553075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:10.553084 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:10.553105 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:10.553161 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:10.553181 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:10.553299 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:10.553385 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 02:30:10.553450 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:30:10.557113 11290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44495 } }
I20260808 02:30:10.557449 11290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:10.557538 11290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cc67f9ae2c4f80bddf0e5d5a7b9966, State: Initialized, Role: FOLLOWER
I20260808 02:30:10.558023 11290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44495 } }
I20260808 02:30:10.558243 11290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:10.558344 11290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:10.558476 11290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:10.560948 11290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44495 } }
I20260808 02:30:10.561407 11290 leader_election.cc:302] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39cc67f9ae2c4f80bddf0e5d5a7b9966; no voters:
I20260808 02:30:10.561923 11290 leader_election.cc:288] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:10.562088 11300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:10.562865 11300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [term 1 LEADER]: Becoming Leader. State: Replica: 39cc67f9ae2c4f80bddf0e5d5a7b9966, State: Running, Role: LEADER
I20260808 02:30:10.563429 11300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44495 } }
I20260808 02:30:10.564182 11290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:10.566900 11303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39cc67f9ae2c4f80bddf0e5d5a7b9966. Latest consensus state: current_term: 1 leader_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44495 } } }
I20260808 02:30:10.567651 11303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:10.566926 11301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cc67f9ae2c4f80bddf0e5d5a7b9966" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44495 } } }
I20260808 02:30:10.568364 11301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:10.569945 11310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:10.577371 11310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:10.588325 11310 catalog_manager.cc:1347] Generated new cluster ID: bc27b0292c124fa1bf1d76c06bf50ed6
I20260808 02:30:10.588495 11310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:10.614269 11310 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:10.614823 11310 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:10.626268 11310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39cc67f9ae2c4f80bddf0e5d5a7b9966: Generated new TSK 0
I20260808 02:30:10.627233 11310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:10.640021 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11215
W20260808 02:30:10.641396 11284 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49647
W20260808 02:30:10.642241 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:30:10.642424 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11215
2026-08-08T02:30:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2137 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:10 dist-test-slave-pgkr krb5kdc[11322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:10 dist-test-slave-pgkr krb5kdc[11322](info): set up 2 sockets
Aug 08 02:30:10 dist-test-slave-pgkr krb5kdc[11322](info): commencing operation
krb5kdc: starting...
W20260808 02:30:11.831432 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:12 dist-test-slave-pgkr krb5kdc[11322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:12.199723 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44937
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:44937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:12.557253 11337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:12.557535 11337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:12.557737 11337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:12.565039 11337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:12.565141 11337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:12.565188 11337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:12.565295 11337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:12.565362 11337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:12.575223 11337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:44937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:12.576690 11337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:12.578029 11337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:12.589792 11343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:12.589844 11346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:12.589869 11344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:12.590957 11337 server_base.cc:1034] running on GCE node
I20260808 02:30:12.591766 11337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:12.593094 11337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:12.594295 11337 hybrid_clock.cc:648] HybridClock initialized: now 1786156212594245 us; error 41 us; skew 500 ppm
I20260808 02:30:12.594647 11337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:12 dist-test-slave-pgkr krb5kdc[11322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156212, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:12.597834 11337 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:12.599411 11337 webserver.cc:460] Webserver started at http://127.19.231.62:43039/ using document root <none> and password file <none>
I20260808 02:30:12.600121 11337 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:12.600261 11337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:12.600592 11337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:12.603741 11337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "465551b6f2e74470ae1a2e700fa3b8b4"
format_stamp: "Formatted at 2026-08-08 02:30:12 on dist-test-slave-pgkr"
server_key: "9e461fde5f95b732f5d683c448e8616c"
server_key_iv: "d06eddc855df84b5137b6ffb00124ca2"
server_key_version: "encryptionkey@0"
I20260808 02:30:12.604610 11337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "465551b6f2e74470ae1a2e700fa3b8b4"
format_stamp: "Formatted at 2026-08-08 02:30:12 on dist-test-slave-pgkr"
server_key: "9e461fde5f95b732f5d683c448e8616c"
server_key_iv: "d06eddc855df84b5137b6ffb00124ca2"
server_key_version: "encryptionkey@0"
I20260808 02:30:12.610098 11337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 02:30:12.613793 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:12.615245 11337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 02:30:12.615435 11337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "465551b6f2e74470ae1a2e700fa3b8b4"
format_stamp: "Formatted at 2026-08-08 02:30:12 on dist-test-slave-pgkr"
server_key: "9e461fde5f95b732f5d683c448e8616c"
server_key_iv: "d06eddc855df84b5137b6ffb00124ca2"
server_key_version: "encryptionkey@0"
I20260808 02:30:12.615653 11337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:12.643366 11337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:12.648494 11337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:12.648808 11337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:12.669461 11337 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:44937
I20260808 02:30:12.669468 11409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:44937 every 8 connection(s)
I20260808 02:30:12.670985 11337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:12.674853 11410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:12.675490 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11337
I20260808 02:30:12.675678 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:12.676277 20380 external_mini_cluster.cc:1428] Setting key b46c35f475bf9d18dffca9ee62c24b46
I20260808 02:30:12.689770 11410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4: Bootstrap starting.
Aug 08 02:30:12 dist-test-slave-pgkr krb5kdc[11322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:12.693113 11410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:12.694545 11410 log.cc:824] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:12.698042 11410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4: No bootstrap required, opened a new log
I20260808 02:30:12.701747 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:12.678212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51901 (local address 127.19.231.62:44937)
0808 02:30:12.679096 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:12.679112 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:12.679167 (+ 55us) server_negotiation.cc:407] Connection header received
0808 02:30:12.681739 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:12.681769 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:12.682314 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:12.682803 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:12.684151 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:12.686195 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:12.687648 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:12.688706 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:12.688897 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:12.693098 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:12.693111 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:12.693131 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:12.693181 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:12.697162 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:12.697849 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:12.697857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:12.697864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:12.698094 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:12.698500 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:12.698507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:12.698512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:12.699241 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:12.699548 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:12.699855 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 02:30:12.700335 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":377,"threads_started":1}
I20260808 02:30:12.705611 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:12.707773 11410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44937 } }
I20260808 02:30:12.708144 11410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:12.708267 11410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465551b6f2e74470ae1a2e700fa3b8b4, State: Initialized, Role: FOLLOWER
I20260808 02:30:12.708899 11410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44937 } }
I20260808 02:30:12.709146 11410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:12.709326 11410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:12.709497 11410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:12.712248 11410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44937 } }
I20260808 02:30:12.712762 11410 leader_election.cc:302] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 465551b6f2e74470ae1a2e700fa3b8b4; no voters:
I20260808 02:30:12.713502 11410 leader_election.cc:288] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:12.713629 11421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:12.714831 11421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [term 1 LEADER]: Becoming Leader. State: Replica: 465551b6f2e74470ae1a2e700fa3b8b4, State: Running, Role: LEADER
I20260808 02:30:12.715456 11421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44937 } }
I20260808 02:30:12.716039 11410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:12.718962 11424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44937 } } }
I20260808 02:30:12.719455 11424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:12.722193 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 465551b6f2e74470ae1a2e700fa3b8b4. Latest consensus state: current_term: 1 leader_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465551b6f2e74470ae1a2e700fa3b8b4" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 44937 } } }
I20260808 02:30:12.722518 11428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:12.722679 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:12.728281 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:12.710309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44465 (local address 127.19.231.62:44937)
0808 02:30:12.710574 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:12.710583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:12.711180 (+ 597us) server_negotiation.cc:407] Connection header received
0808 02:30:12.711316 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:12.711322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:12.711398 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:12.711507 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:12.713027 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:12.713947 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:12.715638 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:12.716350 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:12.716494 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:12.719735 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:12.719742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:12.719746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:12.719789 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:12.723733 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:12.724672 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:12.724680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:12.724683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:12.724874 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:12.725481 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:12.725489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:12.725495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:12.726148 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:12.726502 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:12.727641 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0808 02:30:12.727874 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:30:12.730792 11428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:12.740520 11428 catalog_manager.cc:1347] Generated new cluster ID: 61a313b164894f71810a498489d83ed7
I20260808 02:30:12.740653 11428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:12.770503 11428 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:12.771934 11428 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:12.778039 11428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 465551b6f2e74470ae1a2e700fa3b8b4: Generated new TSK 0
I20260808 02:30:12.778899 11428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:12.793599 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11337
I20260808 02:30:12.794847 11405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:12.890947 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11337
2026-08-08T02:30:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:13 dist-test-slave-pgkr krb5kdc[11446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:13 dist-test-slave-pgkr krb5kdc[11446](info): set up 2 sockets
Aug 08 02:30:13 dist-test-slave-pgkr krb5kdc[11446](info): commencing operation
krb5kdc: starting...
W20260808 02:30:14.087608 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:14 dist-test-slave-pgkr krb5kdc[11446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:14.462488 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45053
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:45053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:14.837109 11461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:14.837437 11461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:14.837607 11461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:14.844978 11461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:14.845130 11461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:14.845219 11461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:14.845335 11461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:14.845429 11461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:14.855612 11461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:45053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:45053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:14.857028 11461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:14.858423 11461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:14.870779 11466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:14.870850 11470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:14.870855 11468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:14.871982 11461 server_base.cc:1034] running on GCE node
I20260808 02:30:14.872782 11461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:14.874245 11461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:14.875479 11461 hybrid_clock.cc:648] HybridClock initialized: now 1786156214875420 us; error 83 us; skew 500 ppm
I20260808 02:30:14.875850 11461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:14 dist-test-slave-pgkr krb5kdc[11446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156214, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:14.879150 11461 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:14.880672 11461 webserver.cc:460] Webserver started at http://127.19.231.62:44239/ using document root <none> and password file <none>
I20260808 02:30:14.881460 11461 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:14.881632 11461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:14.881958 11461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:14.885109 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "3411b0dde31c4d649bd4df423b6833d3"
format_stamp: "Formatted at 2026-08-08 02:30:14 on dist-test-slave-pgkr"
server_key: "b34efa4545db4722a87055e188b9c1a9"
server_key_iv: "9eb5d5b4b765a17afc8769c6913dd560"
server_key_version: "encryptionkey@0"
I20260808 02:30:14.885916 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "3411b0dde31c4d649bd4df423b6833d3"
format_stamp: "Formatted at 2026-08-08 02:30:14 on dist-test-slave-pgkr"
server_key: "b34efa4545db4722a87055e188b9c1a9"
server_key_iv: "9eb5d5b4b765a17afc8769c6913dd560"
server_key_version: "encryptionkey@0"
I20260808 02:30:14.891433 11461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 02:30:14.895020 11478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:14.896409 11461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 02:30:14.896591 11461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "3411b0dde31c4d649bd4df423b6833d3"
format_stamp: "Formatted at 2026-08-08 02:30:14 on dist-test-slave-pgkr"
server_key: "b34efa4545db4722a87055e188b9c1a9"
server_key_iv: "9eb5d5b4b765a17afc8769c6913dd560"
server_key_version: "encryptionkey@0"
I20260808 02:30:14.896802 11461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:14.908128 11461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:14.912420 11461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:14.912714 11461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:14.932914 11461 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:45053
I20260808 02:30:14.932955 11532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:45053 every 8 connection(s)
I20260808 02:30:14.934430 11461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:14.938515 11533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:14.939908 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11461
I20260808 02:30:14.940094 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:14.940531 20380 external_mini_cluster.cc:1428] Setting key 9964d06f6ff16d08825a7fcba293eb83
I20260808 02:30:14.953433 11533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3: Bootstrap starting.
I20260808 02:30:14.955467 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:14.942482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49041 (local address 127.19.231.62:45053)
0808 02:30:14.943461 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:14.943476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:14.943525 (+ 49us) server_negotiation.cc:407] Connection header received
0808 02:30:14.945397 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:14.945425 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:14.945945 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:14.946387 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:14.946914 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:14.948791 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:14.950143 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:14.951302 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:14.951483 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:14.952615 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:14.952625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:14.952643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:14.953331 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:14.953432 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:14.953465 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:14.953594 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:14.953872 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 02:30:14.954133 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":450,"threads_started":1}
I20260808 02:30:14.957396 11533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:14.958700 11533 log.cc:824] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:14.959374 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:14.961428 11533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3: No bootstrap required, opened a new log
I20260808 02:30:14.969835 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:14.963669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49359 (local address 127.19.231.62:45053)
0808 02:30:14.963879 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:14.963887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:14.964274 (+ 387us) server_negotiation.cc:407] Connection header received
0808 02:30:14.964461 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:14.964466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:14.964536 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:14.964651 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:14.965154 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:14.965851 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:14.967325 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:14.967798 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:14.967972 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:14.969101 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:14.969106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:14.969109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:14.969125 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:14.969162 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:14.969177 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:14.969291 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:14.969460 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 02:30:14.969549 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 02:30:14.969928 11533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3411b0dde31c4d649bd4df423b6833d3" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45053 } }
I20260808 02:30:14.970383 11533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:14.970523 11533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3411b0dde31c4d649bd4df423b6833d3, State: Initialized, Role: FOLLOWER
I20260808 02:30:14.971098 11533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3411b0dde31c4d649bd4df423b6833d3" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45053 } }
I20260808 02:30:14.971370 11533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:14.971525 11533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:14.971669 11533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:14.973886 11533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3411b0dde31c4d649bd4df423b6833d3" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45053 } }
I20260808 02:30:14.974431 11533 leader_election.cc:302] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3411b0dde31c4d649bd4df423b6833d3; no voters:
I20260808 02:30:14.975121 11533 leader_election.cc:288] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:14.975238 11544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:14.975973 11544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3411b0dde31c4d649bd4df423b6833d3, State: Running, Role: LEADER
I20260808 02:30:14.976511 11544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3411b0dde31c4d649bd4df423b6833d3" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45053 } }
I20260808 02:30:14.977201 11533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:14.979424 11545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3411b0dde31c4d649bd4df423b6833d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3411b0dde31c4d649bd4df423b6833d3" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45053 } } }
I20260808 02:30:14.979884 11545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:14.982438 11552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:14.982388 11546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3411b0dde31c4d649bd4df423b6833d3. Latest consensus state: current_term: 1 leader_uuid: "3411b0dde31c4d649bd4df423b6833d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3411b0dde31c4d649bd4df423b6833d3" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 45053 } } }
I20260808 02:30:14.982908 11546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:14.987447 11552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:14.988183 11562 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 02:30:14.996228 11552 catalog_manager.cc:1347] Generated new cluster ID: ce07277922074f758393182cbbc852e2
I20260808 02:30:14.996348 11552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:15.022604 11552 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:15.024699 11552 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:15.037627 11552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3411b0dde31c4d649bd4df423b6833d3: Generated new TSK 0
I20260808 02:30:15.038728 11552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:15.047957 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11461
W20260808 02:30:15.049743 11528 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49041
W20260808 02:30:15.050472 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11461: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:30:15.050673 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11461
2026-08-08T02:30:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2159 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:15 dist-test-slave-pgkr krb5kdc[11567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:15 dist-test-slave-pgkr krb5kdc[11567](info): set up 2 sockets
Aug 08 02:30:15 dist-test-slave-pgkr krb5kdc[11567](info): commencing operation
krb5kdc: starting...
W20260808 02:30:16.275040 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.006s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:16 dist-test-slave-pgkr krb5kdc[11567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:16.634405 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40649
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:40649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:16.988504 11583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:16.988852 11583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:16.989017 11583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:16.996359 11583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:16.996498 11583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:16.996603 11583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:16.996752 11583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:16.996841 11583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:17.006924 11583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:40649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:17.008333 11583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:17.009712 11583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:17.022869 11589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:17.022903 11591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:17.023663 11588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:17.023690 11583 server_base.cc:1034] running on GCE node
I20260808 02:30:17.024557 11583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:17.026031 11583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:17.027268 11583 hybrid_clock.cc:648] HybridClock initialized: now 1786156217027208 us; error 52 us; skew 500 ppm
I20260808 02:30:17.027671 11583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:17 dist-test-slave-pgkr krb5kdc[11567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156217, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:17.035660 11583 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:17.037381 11583 webserver.cc:460] Webserver started at http://127.19.231.62:34561/ using document root <none> and password file <none>
I20260808 02:30:17.038215 11583 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:17.038410 11583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:17.038825 11583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:17.042045 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "36a512e898cc4a7d9648743bdcd74217"
format_stamp: "Formatted at 2026-08-08 02:30:17 on dist-test-slave-pgkr"
server_key: "d48f2ef3b78f1ba1295e8ed8cb780cf1"
server_key_iv: "e920782c459cf4cef351e26346d75f27"
server_key_version: "encryptionkey@0"
I20260808 02:30:17.042955 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "36a512e898cc4a7d9648743bdcd74217"
format_stamp: "Formatted at 2026-08-08 02:30:17 on dist-test-slave-pgkr"
server_key: "d48f2ef3b78f1ba1295e8ed8cb780cf1"
server_key_iv: "e920782c459cf4cef351e26346d75f27"
server_key_version: "encryptionkey@0"
I20260808 02:30:17.048725 11583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:30:17.052482 11598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:17.054111 11583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 02:30:17.054301 11583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "36a512e898cc4a7d9648743bdcd74217"
format_stamp: "Formatted at 2026-08-08 02:30:17 on dist-test-slave-pgkr"
server_key: "d48f2ef3b78f1ba1295e8ed8cb780cf1"
server_key_iv: "e920782c459cf4cef351e26346d75f27"
server_key_version: "encryptionkey@0"
I20260808 02:30:17.054507 11583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:17.060652 11583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:17.064901 11583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:17.065207 11583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:17.085198 11583 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:40649
I20260808 02:30:17.085235 11651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:40649 every 8 connection(s)
I20260808 02:30:17.086856 11583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:17.089146 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11583
I20260808 02:30:17.089332 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:17.089843 20380 external_mini_cluster.cc:1428] Setting key fea504d99da5318b0374a4f2e15226db
I20260808 02:30:17.091162 11652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:17.098709 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:17.091675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41641 (local address 127.19.231.62:40649)
0808 02:30:17.092531 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:17.092549 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:17.092613 (+ 64us) server_negotiation.cc:407] Connection header received
0808 02:30:17.094651 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:17.094680 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:17.095135 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:17.095591 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:17.096623 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:17.096632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:17.096649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:17.096669 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:17.096772 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:17.096799 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:17.097022 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:17.097161 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 02:30:17.097362 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":331,"threads_started":1}
I20260808 02:30:17.102316 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:17.106007 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217: Bootstrap starting.
I20260808 02:30:17.110337 11652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:17.110737 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:17.107739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56737 (local address 127.19.231.62:40649)
0808 02:30:17.107952 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:17.107959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:17.108529 (+ 570us) server_negotiation.cc:407] Connection header received
0808 02:30:17.108713 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:17.108717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:17.108764 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:17.108842 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:17.109803 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:17.109807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:17.109810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 02:30:17.109824 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:17.109863 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 02:30:17.109877 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:17.109958 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:17.110101 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 02:30:17.110146 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:30:17.111613 11652 log.cc:824] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:17.114373 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217: No bootstrap required, opened a new log
I20260808 02:30:17.121616 11652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a512e898cc4a7d9648743bdcd74217" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40649 } }
I20260808 02:30:17.122006 11652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 02:30:17.122136 11652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a512e898cc4a7d9648743bdcd74217, State: Initialized, Role: FOLLOWER
I20260808 02:30:17.122745 11652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a512e898cc4a7d9648743bdcd74217" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40649 } }
I20260808 02:30:17.122982 11652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 02:30:17.123129 11652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 02:30:17.123299 11652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:17.125488 11652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a512e898cc4a7d9648743bdcd74217" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40649 } }
I20260808 02:30:17.125976 11652 leader_election.cc:302] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36a512e898cc4a7d9648743bdcd74217; no voters:
I20260808 02:30:17.126724 11652 leader_election.cc:288] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 02:30:17.126817 11662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:17.127630 11662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [term 1 LEADER]: Becoming Leader. State: Replica: 36a512e898cc4a7d9648743bdcd74217, State: Running, Role: LEADER
I20260808 02:30:17.128248 11662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a512e898cc4a7d9648743bdcd74217" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40649 } }
I20260808 02:30:17.128950 11652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:17.134683 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a512e898cc4a7d9648743bdcd74217. Latest consensus state: current_term: 1 leader_uuid: "36a512e898cc4a7d9648743bdcd74217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a512e898cc4a7d9648743bdcd74217" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40649 } } }
I20260808 02:30:17.134683 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36a512e898cc4a7d9648743bdcd74217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a512e898cc4a7d9648743bdcd74217" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 40649 } } }
I20260808 02:30:17.135300 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:17.135300 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:17.139078 11670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:17.144692 11670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:17.154841 11670 catalog_manager.cc:1347] Generated new cluster ID: bf7c751dd48e41dd9063058f350ff0fc
I20260808 02:30:17.155004 11670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:17.174223 11670 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:17.174847 11670 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:17.184404 11670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36a512e898cc4a7d9648743bdcd74217: Generated new TSK 0
I20260808 02:30:17.185405 11670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:17.200989 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11583
W20260808 02:30:17.202133 11647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41641
W20260808 02:30:17.202920 20380 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 02:30:17.203104 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11583
2026-08-08T02:30:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2152 ms)
[----------] 18 tests from AuthTokenIssuingTest (39276 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:17 dist-test-slave-pgkr krb5kdc[11685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:17 dist-test-slave-pgkr krb5kdc[11685](info): set up 2 sockets
Aug 08 02:30:17 dist-test-slave-pgkr krb5kdc[11685](info): commencing operation
krb5kdc: starting...
W20260808 02:30:18.435308 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:18 dist-test-slave-pgkr krb5kdc[11685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:18.798579 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37481
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37481,127.19.231.61:35953,127.19.231.60:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:19.156960 11701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:30:19.157328 11701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:19.157449 11701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:19.157598 11701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:19.164822 11701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:19.164918 11701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:19.165037 11701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:19.165104 11701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:19.165189 11701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:19.175034 11701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37481,127.19.231.61:35953,127.19.231.60:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:19.176445 11701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:19.177870 11701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:19.190246 11708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:19.190212 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:19.190253 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:19.191351 11701 server_base.cc:1034] running on GCE node
I20260808 02:30:19.192198 11701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:19.193643 11701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:19.194883 11701 hybrid_clock.cc:648] HybridClock initialized: now 1786156219194825 us; error 34 us; skew 500 ppm
I20260808 02:30:19.195207 11701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:19 dist-test-slave-pgkr krb5kdc[11685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:19.198523 11701 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:19.200210 11701 webserver.cc:460] Webserver started at http://127.19.231.62:41783/ using document root <none> and password file <none>
I20260808 02:30:19.200930 11701 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:19.201062 11701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:19.201395 11701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:19.204581 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "2ec25333aa4c46c09005f82472b17cf7"
format_stamp: "Formatted at 2026-08-08 02:30:19 on dist-test-slave-pgkr"
server_key: "398ead9a726c083f362589306c23cc3d"
server_key_iv: "1009d17c466c3e68dec952dbe3548b7a"
server_key_version: "encryptionkey@0"
I20260808 02:30:19.205408 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "2ec25333aa4c46c09005f82472b17cf7"
format_stamp: "Formatted at 2026-08-08 02:30:19 on dist-test-slave-pgkr"
server_key: "398ead9a726c083f362589306c23cc3d"
server_key_iv: "1009d17c466c3e68dec952dbe3548b7a"
server_key_version: "encryptionkey@0"
I20260808 02:30:19.210960 11701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 02:30:19.214578 11718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:19.216048 11701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:30:19.216240 11701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "2ec25333aa4c46c09005f82472b17cf7"
format_stamp: "Formatted at 2026-08-08 02:30:19 on dist-test-slave-pgkr"
server_key: "398ead9a726c083f362589306c23cc3d"
server_key_iv: "1009d17c466c3e68dec952dbe3548b7a"
server_key_version: "encryptionkey@0"
I20260808 02:30:19.216465 11701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:19.254771 11701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:19.259740 11701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:19.260036 11701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:19.280782 11701 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:37481
I20260808 02:30:19.280836 11773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:37481 every 8 connection(s)
I20260808 02:30:19.282198 11701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:19.285544 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11701
I20260808 02:30:19.285808 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:19.286792 20380 external_mini_cluster.cc:1428] Setting key 13a487b0584622151c0fa31a4609e617
I20260808 02:30:19.286926 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:19.298276 11774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:19 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.231.61@KRBTEST.COM; defaulting to no policy
I20260808 02:30:19.325647 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.303834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.19.231.62:37481)
0808 02:30:19.304691 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:19.304701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:19.304764 (+ 63us) server_negotiation.cc:407] Connection header received
0808 02:30:19.306166 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:19.306172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:19.306446 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:19.306721 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:19.308402 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.309787 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.312134 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.312760 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.312908 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:19.317249 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:19.317257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:19.317271 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:19.317302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:19.320561 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:19.321455 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:19.321462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:19.321465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:19.321621 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:19.322068 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:19.322072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:19.322074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:19.322687 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:19.322910 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:19.323517 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 02:30:19.323908 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":561,"thread_start_us":343,"threads_started":1}
I20260808 02:30:19.325671 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.302408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37481 (local address 127.0.0.1:52552)
0808 02:30:19.303412 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.303460 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:19.305739 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:19.306767 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:19.306777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:19.307628 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:19.308160 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.308188 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.309958 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.309966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:19.311972 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.311989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.313019 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.313029 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:19.313285 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:19.314024 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:19.314059 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:19.316996 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:19.320706 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:19.320719 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:19.320732 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:19.321306 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:19.321740 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:19.321747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:19.321751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:19.321937 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:19.322839 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:19.322847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:19.323115 (+ 268us) client_negotiation.cc:769] Sending connection context
0808 02:30:19.323381 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 02:30:19.324047 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":161,"thread_start_us":451,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.61@KRBTEST.COM" created.
I20260808 02:30:19.329270 11774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:19.330415 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.330002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44144)
0808 02:30:19.330171 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.330297 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 02:30:19.330768 11720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
I20260808 02:30:19.332401 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.332027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44146)
0808 02:30:19.332174 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.332243 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 02:30:19.332600 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:35953: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab.
Entry for principal kudu/127.19.231.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab.
I20260808 02:30:19.368120 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:35953
--webserver_interface=127.19.231.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab
--principal=kudu/127.19.231.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37481,127.19.231.61:35953,127.19.231.60:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:30:19.381970 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } attempt: 1
I20260808 02:30:19.383435 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.382814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44148)
0808 02:30:19.383088 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.383261 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 02:30:19.384622 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.384250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44154)
0808 02:30:19.384401 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.384486 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:30:19.384883 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:35953: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
I20260808 02:30:19.453164 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } attempt: 2
I20260808 02:30:19.454417 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.453929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44156)
0808 02:30:19.454186 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.454280 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 02:30:19.455461 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.455125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44160)
0808 02:30:19.455265 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.455331 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 02:30:19.455724 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:35953: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
I20260808 02:30:19.547000 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } attempt: 3
I20260808 02:30:19.548331 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.547864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44176)
0808 02:30:19.548083 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.548181 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 02:30:19.549386 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.549044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44178)
0808 02:30:19.549178 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.549249 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:30:19.549640 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:35953: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
I20260808 02:30:19.692893 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } attempt: 4
I20260808 02:30:19.694217 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.693639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44188)
0808 02:30:19.693972 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.694064 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260808 02:30:19.695335 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.695013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44198)
0808 02:30:19.695131 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.695224 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 02:30:19.695640 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:35953: Network error: Client connection negotiation failed: client connection to 127.19.231.61:35953: connect: Connection refused (error 111)
W20260808 02:30:19.726300 11781 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:30:19.726679 11781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:19.726926 11781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:19.727097 11781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:19.734902 11781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:19.735013 11781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:19.735137 11781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:19.735220 11781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:19.735358 11781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:19.746158 11781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37481,127.19.231.61:35953,127.19.231.60:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab
--principal=kudu/127.19.231.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:35953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.231.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:19.747684 11781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:19.749034 11781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:19.760742 11789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:19.760812 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:19.760782 11792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:19.761873 11781 server_base.cc:1034] running on GCE node
I20260808 02:30:19.762782 11781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:19.764197 11781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:19.765452 11781 hybrid_clock.cc:648] HybridClock initialized: now 1786156219765383 us; error 42 us; skew 500 ppm
I20260808 02:30:19.765818 11781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:19 dist-test-slave-pgkr krb5kdc[11685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:19.769012 11781 init.cc:377] Logged in from keytab as kudu/127.19.231.61@KRBTEST.COM (short username kudu)
I20260808 02:30:19.770521 11781 webserver.cc:460] Webserver started at http://127.19.231.61:44797/ using document root <none> and password file <none>
I20260808 02:30:19.771311 11781 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:19.771441 11781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:19.771790 11781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:19.775017 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data/instance:
uuid: "b446ac56fdc84bfcbae4ae08e6bc1063"
format_stamp: "Formatted at 2026-08-08 02:30:19 on dist-test-slave-pgkr"
server_key: "b96f270db56d0d423e0daf20b72f7e4b"
server_key_iv: "b444b47a7bbe6fe01fbc3f60c6e3266c"
server_key_version: "encryptionkey@0"
I20260808 02:30:19.775851 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/wal/instance:
uuid: "b446ac56fdc84bfcbae4ae08e6bc1063"
format_stamp: "Formatted at 2026-08-08 02:30:19 on dist-test-slave-pgkr"
server_key: "b96f270db56d0d423e0daf20b72f7e4b"
server_key_iv: "b444b47a7bbe6fe01fbc3f60c6e3266c"
server_key_version: "encryptionkey@0"
I20260808 02:30:19.781260 11781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 02:30:19.784972 11799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:19.786541 11781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 02:30:19.786762 11781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/wal
uuid: "b446ac56fdc84bfcbae4ae08e6bc1063"
format_stamp: "Formatted at 2026-08-08 02:30:19 on dist-test-slave-pgkr"
server_key: "b96f270db56d0d423e0daf20b72f7e4b"
server_key_iv: "b444b47a7bbe6fe01fbc3f60c6e3266c"
server_key_version: "encryptionkey@0"
I20260808 02:30:19.786986 11781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:19.813469 11781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:19.818399 11781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:19.818755 11781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:19.838256 11781 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.61:35953
I20260808 02:30:19.838264 11856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.61:35953 every 8 connection(s)
I20260808 02:30:19.839774 11781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
I20260808 02:30:19.843837 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:19.844128 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11781
I20260808 02:30:19.844336 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-1/wal/instance
I20260808 02:30:19.844944 20380 external_mini_cluster.cc:1428] Setting key 93450d279f4727681427850a9d055461
I20260808 02:30:19.854159 11857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:19 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:19.879622 11859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.857733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.19.231.62:37481)
0808 02:30:19.858323 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:19.858331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:19.859484 (+ 1153us) server_negotiation.cc:407] Connection header received
0808 02:30:19.862234 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:19.862242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:19.862359 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:19.862562 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:19.864085 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.865386 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.866957 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.867756 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.867918 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:19.871713 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:19.871723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:19.871731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:19.871799 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:19.875854 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:19.876678 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:19.876688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:19.876694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:19.876871 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:19.877299 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:19.877308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:19.877314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:19.877828 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:19.878042 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:19.878759 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 02:30:19.879048 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":298,"threads_started":1}
I20260808 02:30:19.880319 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.858379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37481 (local address 127.0.0.1:52564)
0808 02:30:19.859279 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.859335 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:19.861941 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:19.862758 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:19.862766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:19.863307 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:19.863852 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.863868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.865550 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.865553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:19.866812 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.866824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.867969 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.867974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:19.868158 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:19.868885 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:19.868907 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:19.871426 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:19.876007 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:19.876014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:19.876028 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:19.876511 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:19.876984 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:19.876987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:19.876989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:19.877161 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:19.877971 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:19.877977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:19.878327 (+ 350us) client_negotiation.cc:769] Sending connection context
0808 02:30:19.878555 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 02:30:19.879115 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":370,"threads_started":1}
I20260808 02:30:19.882162 11857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.231.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.60@KRBTEST.COM" created.
Aug 08 02:30:19 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:30:19.898514 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.883002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44206)
0808 02:30:19.883248 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.883271 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:19.883419 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:19.884435 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:19.884439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:19.884823 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:19.885048 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.885058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.886470 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.886478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:19.887641 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.887658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.888464 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.888469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:19.888709 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:19.889300 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:19.889317 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:19.891519 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:19.895983 (+ 4464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:19.895987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:19.895989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:19.896375 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:19.896785 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:19.896788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:19.896791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:19.896930 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:19.897703 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:19.897707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:19.897836 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 02:30:19.897973 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 02:30:19.898219 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":7}
I20260808 02:30:19.899103 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.883281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.19.231.61:35953)
0808 02:30:19.884016 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:19.884022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:19.884049 (+ 27us) server_negotiation.cc:407] Connection header received
0808 02:30:19.884093 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:19.884099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:19.884275 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:19.884397 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:19.885215 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.886304 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.887798 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.888323 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.888457 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:19.891754 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:19.891769 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:19.891788 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:19.891831 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:19.895823 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:19.896509 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:19.896520 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:19.896525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:19.896676 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:19.897033 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:19.897040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:19.897046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:19.897541 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:19.897772 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:19.898155 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 02:30:19.898454 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":588,"thread_start_us":439,"threads_started":1}
I20260808 02:30:19.901829 11857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:19.903208 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.902501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39500)
0808 02:30:19.902831 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.902987 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260808 02:30:19.903513 11800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:19.905213 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.904742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39510)
0808 02:30:19.904938 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.905056 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:30:19.905441 11857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab.
Entry for principal kudu/127.19.231.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab.
I20260808 02:30:19.929250 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:40701
--webserver_interface=127.19.231.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab
--principal=kudu/127.19.231.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:37481,127.19.231.61:35953,127.19.231.60:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:30:19.954834 11857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 1
I20260808 02:30:19.956063 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.955590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39512)
0808 02:30:19.955781 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.955893 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:30:19.957067 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.956736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39518)
0808 02:30:19.956881 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.956963 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 02:30:19.957293 11857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:19.994915 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } attempt: 5
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:30:20.009637 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.995797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.19.231.61:35953)
0808 02:30:19.995974 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:19.995978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:19.996037 (+ 59us) server_negotiation.cc:407] Connection header received
0808 02:30:19.996290 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:19.996295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:19.996361 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:19.996447 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:19.997511 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.998219 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.999905 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.000373 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.000470 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.004293 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.004299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.004302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.004335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.007049 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.007826 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.007829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.007831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.007950 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.008372 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.008375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.008377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.008646 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.008759 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.009244 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.009415 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:30:20.009768 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:19.995686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44210)
0808 02:30:19.995932 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:19.995959 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:19.996137 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:19.996522 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:19.996528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:19.997105 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:19.997361 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.997376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:19.998358 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:19.998365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:19.999746 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:19.999762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.000526 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.000532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.000746 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.001421 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:20.001444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:20.004091 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:20.007181 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.007186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.007190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.007716 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.008067 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.008071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.008075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.008265 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.008761 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:20.008766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:20.008923 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 02:30:20.009143 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 02:30:20.009423 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 02:30:20.010852 11774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:20.011952 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.011489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39532)
0808 02:30:20.011666 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.011762 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 02:30:20.013059 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.012684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39538)
0808 02:30:20.012834 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.012930 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 02:30:20.013298 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:20.025600 11857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 2
I20260808 02:30:20.026795 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.026279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39540)
0808 02:30:20.026469 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.026568 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:30:20.028282 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.027725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39554)
0808 02:30:20.028012 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.028120 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260808 02:30:20.028613 11857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:20.064569 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 1
I20260808 02:30:20.065878 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.065310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39560)
0808 02:30:20.065549 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.065668 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 02:30:20.066931 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.066560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39566)
0808 02:30:20.066728 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.066810 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 02:30:20.067193 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:20.119870 11857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 3
I20260808 02:30:20.121179 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.120629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39570)
0808 02:30:20.120851 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.120965 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 02:30:20.122359 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.121878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39580)
0808 02:30:20.122129 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.122223 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260808 02:30:20.122594 11857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:20.135432 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 2
I20260808 02:30:20.136649 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.136112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39590)
0808 02:30:20.136347 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.136452 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 02:30:20.137751 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.137314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39598)
0808 02:30:20.137486 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.137601 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 02:30:20.138051 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:20.244316 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 3
I20260808 02:30:20.245610 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.245040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39610)
0808 02:30:20.245294 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.245404 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 02:30:20.246768 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.246302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39614)
0808 02:30:20.246514 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.246606 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260808 02:30:20.246999 11774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
I20260808 02:30:20.265882 11857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 4
I20260808 02:30:20.267231 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.266591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39624)
0808 02:30:20.266890 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.267023 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 02:30:20.268262 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.267880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39640)
0808 02:30:20.268033 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.268126 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 02:30:20.268479 11857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:40701: Network error: Client connection negotiation failed: client connection to 127.19.231.60:40701: connect: Connection refused (error 111)
W20260808 02:30:20.299603 11865 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:30:20.299908 11865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:20.299981 11865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:20.300071 11865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:20.307461 11865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:20.307554 11865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:20.307613 11865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:20.307665 11865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:20.307732 11865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:20.317610 11865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:37481,127.19.231.61:35953,127.19.231.60:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab
--principal=kudu/127.19.231.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.231.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:20.318907 11865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:20.320209 11865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:20.332154 11871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:20.332240 11870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:20.332405 11873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:20.332974 11865 server_base.cc:1034] running on GCE node
I20260808 02:30:20.333679 11865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:20.335068 11865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:20.336284 11865 hybrid_clock.cc:648] HybridClock initialized: now 1786156220336203 us; error 49 us; skew 500 ppm
I20260808 02:30:20.336647 11865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156220, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:20.339900 11865 init.cc:377] Logged in from keytab as kudu/127.19.231.60@KRBTEST.COM (short username kudu)
I20260808 02:30:20.341382 11865 webserver.cc:460] Webserver started at http://127.19.231.60:37131/ using document root <none> and password file <none>
I20260808 02:30:20.342054 11865 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:20.342170 11865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:20.342499 11865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:20.345546 11865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data/instance:
uuid: "a9683409235042d1b4bac709d1bacedc"
format_stamp: "Formatted at 2026-08-08 02:30:20 on dist-test-slave-pgkr"
server_key: "177b80f65606099edbf8605b6085de85"
server_key_iv: "4e70c48e51f77cafd8f5c03157a313cd"
server_key_version: "encryptionkey@0"
I20260808 02:30:20.346369 11865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/wal/instance:
uuid: "a9683409235042d1b4bac709d1bacedc"
format_stamp: "Formatted at 2026-08-08 02:30:20 on dist-test-slave-pgkr"
server_key: "177b80f65606099edbf8605b6085de85"
server_key_iv: "4e70c48e51f77cafd8f5c03157a313cd"
server_key_version: "encryptionkey@0"
I20260808 02:30:20.351711 11865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 02:30:20.355229 11881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:20.356670 11865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:30:20.356835 11865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/wal
uuid: "a9683409235042d1b4bac709d1bacedc"
format_stamp: "Formatted at 2026-08-08 02:30:20 on dist-test-slave-pgkr"
server_key: "177b80f65606099edbf8605b6085de85"
server_key_iv: "4e70c48e51f77cafd8f5c03157a313cd"
server_key_version: "encryptionkey@0"
I20260808 02:30:20.357043 11865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:20.378105 11865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:20.382437 11865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:20.382778 11865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:20.402088 11865 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.60:40701
I20260808 02:30:20.402129 11933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.60:40701 every 8 connection(s)
I20260808 02:30:20.403652 11865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
I20260808 02:30:20.406054 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 11865
I20260808 02:30:20.406261 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786155839876607-20380-0/minicluster-data/master-2/wal/instance
I20260808 02:30:20.406764 20380 external_mini_cluster.cc:1428] Setting key 3d51aadc7c2c23b4f1d24a714aaff4af
I20260808 02:30:20.407549 11934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:20.417445 11934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:30:20.424232 11774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 4
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156220, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:20.431996 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.408659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.19.231.60:40701)
0808 02:30:20.409434 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.409446 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.409491 (+ 45us) server_negotiation.cc:407] Connection header received
0808 02:30:20.411090 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.411112 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.411499 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.411847 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.413755 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.415241 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.417101 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.417783 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.417917 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.422941 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.422953 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.422970 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.423027 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.427169 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.427970 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.427983 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.427989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.428161 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.428634 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.428641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.428645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.429376 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.429688 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.430222 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.430706 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":371,"threads_started":1}
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:30:20.440065 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.420663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:37481 (local address 127.0.0.1:52572)
0808 02:30:20.421620 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.421663 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:20.421816 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:20.422268 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:20.422274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:20.423248 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:20.423669 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.423685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.425643 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.425648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.427422 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.427444 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.428438 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.428445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.428658 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.429275 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:20.429293 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:20.432386 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:20.436962 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.436977 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.436986 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.437496 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.437869 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.437874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.437878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.438043 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.438850 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:20.438859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:20.439062 (+ 203us) client_negotiation.cc:769] Sending connection context
0808 02:30:20.439253 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 02:30:20.439602 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":58,"thread_start_us":587,"threads_started":1}
I20260808 02:30:20.441076 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.420630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.19.231.62:37481)
0808 02:30:20.421229 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.421235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.421818 (+ 583us) server_negotiation.cc:407] Connection header received
0808 02:30:20.421941 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.421948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.422027 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.422162 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.423842 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.425481 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.427614 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.428242 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.428380 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.432598 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.432604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.432608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.432669 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.436789 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.437624 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.437630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.437634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.437760 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.438157 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.438162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.438165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.438536 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.438766 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.440598 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.440773 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":470,"spinlock_wait_cycles":32128,"thread_start_us":296,"threads_started":1}
I20260808 02:30:20.441396 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.424840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39666)
0808 02:30:20.425024 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.425046 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:20.425153 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:20.426116 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:20.426122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:20.426686 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:20.426878 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.426889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.428584 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.428592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.429920 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.429935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.430907 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.430914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.431118 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.431739 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:20.431763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:20.434590 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:20.438733 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.438738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.438742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.439142 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.439524 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.439530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.439534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.439660 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.440288 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:20.440292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:20.440471 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 02:30:20.440617 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 02:30:20.440921 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1,"spinlock_wait_cycles":1920}
I20260808 02:30:20.441962 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.425003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.19.231.60:40701)
0808 02:30:20.425756 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.425764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.425799 (+ 35us) server_negotiation.cc:407] Connection header received
0808 02:30:20.425846 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.425853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.425941 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.426074 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.427064 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.428482 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.430089 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.430759 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.430877 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.434881 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.434887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.434896 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.434924 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.438545 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.439267 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.439273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.439277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.439412 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.439766 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.439772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.439775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.440135 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.441196 (+ 1061us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.441325 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.441551 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":601,"thread_start_us":394,"threads_started":1}
I20260808 02:30:20.443056 11934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:20.448951 11774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7: Bootstrap starting.
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156220, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:30:20.453598 11774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:20.455135 11774 log.cc:824] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:20.456372 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.435836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.19.231.62:37481)
0808 02:30:20.436490 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.436499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.436539 (+ 40us) server_negotiation.cc:407] Connection header received
0808 02:30:20.436588 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.436593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.436663 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.436772 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.438517 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.439863 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.441649 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.442383 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.443134 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.448385 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.448397 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.448402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.448441 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.452185 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.453000 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.453006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.453010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.453154 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.453618 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.453625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.453628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.454079 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.455443 (+ 1364us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.455563 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.455837 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":274,"threads_started":1}
I20260808 02:30:20.459293 11774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7: No bootstrap required, opened a new log
I20260808 02:30:20.461431 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.443823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.19.231.61:35953)
0808 02:30:20.444002 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.444006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.446887 (+ 2881us) server_negotiation.cc:407] Connection header received
0808 02:30:20.447033 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.447039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.447127 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.447234 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.448108 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.448867 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.450216 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.450708 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.450844 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.454155 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.454161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.454171 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.454208 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.457981 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.458895 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.458901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.458904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.459105 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.459656 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.459660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.459663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.460044 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.460211 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.460775 (+ 564us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.461003 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 02:30:20.461607 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.443878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:35953 (local address 127.0.0.1:44224)
0808 02:30:20.446781 (+ 2903us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.446799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:20.446906 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:20.447277 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:20.447285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:20.447734 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:20.447930 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.447942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.449009 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.449015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.450060 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.450073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.450857 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.450863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.451069 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.451599 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:20.451616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:20.453971 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:20.458145 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.458151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.458154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.458730 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.459252 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.459257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.459261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.459442 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.460192 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:20.460209 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:20.460437 (+ 228us) client_negotiation.cc:769] Sending connection context
0808 02:30:20.460663 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 02:30:20.461078 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784}
I20260808 02:30:20.463200 11934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:20.471174 11774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
I20260808 02:30:20.471623 11774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec25333aa4c46c09005f82472b17cf7, State: Initialized, Role: FOLLOWER
I20260808 02:30:20.472225 11774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:30:20.474944 11945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156220, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:30:20.475378 11945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:20.476547 11774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:20.482779 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.463944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39676)
0808 02:30:20.464487 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.464509 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:20.464623 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:20.465313 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:20.465320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:20.465943 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:20.466181 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.466208 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.467962 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.467969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.470237 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.470259 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.471833 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.471842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.472096 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.472993 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:20.473026 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:20.476035 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:20.479734 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.479740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.479744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.480145 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.480568 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.480574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.480578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.480769 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.481512 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:20.481517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:20.481675 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 02:30:20.481822 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 02:30:20.482344 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":213}
I20260808 02:30:20.483239 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.459166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.19.231.61:35953)
0808 02:30:20.462177 (+ 3011us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.462182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.462205 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:30:20.462262 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.462268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.462340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.464783 (+ 2443us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.464914 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.466043 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.467937 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.468637 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.468762 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.474979 (+ 6217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.474985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.474989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.475026 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.479077 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.480054 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.480060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.480064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.480238 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.480799 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.480807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.480811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.481247 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.481587 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.482262 (+ 675us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.482490 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2904,"thread_start_us":15052,"threads_started":1}
I20260808 02:30:20.483419 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.464296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.19.231.60:40701)
0808 02:30:20.464865 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.464873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.464893 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:30:20.464943 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.464950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.465026 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.465126 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.466380 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.467798 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.470452 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.471325 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.471467 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.476224 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.476228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.476231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.476258 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.479604 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.480265 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.480272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.480277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.480445 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.480902 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.480909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.480914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.481357 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.481547 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.481924 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.483098 (+ 1174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422}
W20260808 02:30:20.490512 11957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:30:20.490751 11957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:30:20.490854 11934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: Bootstrap starting.
I20260808 02:30:20.495285 11934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:20.496773 11934 log.cc:824] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:20.497126 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:20.500691 11746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2ec25333aa4c46c09005f82472b17cf7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52588
I20260808 02:30:20.501121 11746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:30:20.501366 11746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:20.489794 11701 master_runner.cc:186] Error getting master registration for 127.19.231.61:35953: OK, instance_id { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" instance_seqno: 1786156219832839 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:30:20.502852 11701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:30:20.505155 11746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
I20260808 02:30:20.507771 11746 leader_election.cc:288] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [CANDIDATE]: Term 1 election: Requested vote from peers b446ac56fdc84bfcbae4ae08e6bc1063 (127.19.231.61:35953), a9683409235042d1b4bac709d1bacedc (127.19.231.60:40701)
I20260808 02:30:20.508628 11934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: No bootstrap required, opened a new log
W20260808 02:30:20.513144 11724 tablet.cc:2369] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:30:20.522855 11934 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
I20260808 02:30:20.523448 11934 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9683409235042d1b4bac709d1bacedc, State: Initialized, Role: FOLLOWER
I20260808 02:30:20.524113 11934 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
I20260808 02:30:20.530903 11967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.531404 11967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:20.532608 11934 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:20.532671 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ec25333aa4c46c09005f82472b17cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9683409235042d1b4bac709d1bacedc"
I20260808 02:30:20.533224 11909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:20.536973 11909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ec25333aa4c46c09005f82472b17cf7 in term 1.
I20260808 02:30:20.537703 11719 leader_election.cc:302] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ec25333aa4c46c09005f82472b17cf7, a9683409235042d1b4bac709d1bacedc; no voters:
I20260808 02:30:20.538240 11945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:20.542689 11945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [term 1 LEADER]: Becoming Leader. State: Replica: 2ec25333aa4c46c09005f82472b17cf7, State: Running, Role: LEADER
I20260808 02:30:20.543500 11945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
I20260808 02:30:20.552742 11971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec25333aa4c46c09005f82472b17cf7. Latest consensus state: current_term: 1 leader_uuid: "2ec25333aa4c46c09005f82472b17cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.553251 11971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:20.561574 11976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:20.563360 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.526727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.19.231.60:40701)
0808 02:30:20.526949 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.526958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.530634 (+ 3676us) server_negotiation.cc:407] Connection header received
0808 02:30:20.531870 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.531877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.531984 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.532063 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.534379 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.535970 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.547625 (+ 11655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.548592 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.548766 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.554632 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.554651 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.554659 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.554716 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.559357 (+ 4641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.560438 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.560496 (+ 58us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.560503 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.560711 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.561231 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.561235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.561238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.561751 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.562030 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.562723 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.562975 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:30:20.564666 11976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:20.567677 11857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } attempt: 5
I20260808 02:30:20.572279 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.513125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.19.231.61:35953)
0808 02:30:20.513357 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.513364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.528987 (+ 15623us) server_negotiation.cc:407] Connection header received
0808 02:30:20.529904 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.529911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.529997 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.530075 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.542998 (+ 12923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.544192 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.549509 (+ 5317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.550224 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.550336 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.557775 (+ 7439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.557782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.557787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.557832 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.561824 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.567804 (+ 5980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.567810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.567813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.568037 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.569171 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.569177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.569180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.569591 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.569841 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.571620 (+ 1779us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.571883 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:30:20.575037 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.513271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.19.231.62:37481)
0808 02:30:20.513486 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.513492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.525719 (+ 12227us) server_negotiation.cc:407] Connection header received
0808 02:30:20.527736 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.527743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.527840 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.527920 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.546167 (+ 18247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.547446 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.551328 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.552118 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.552239 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.562892 (+ 10653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.562900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.562906 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.562949 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.566918 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.568516 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.568521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.568524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.568678 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.570351 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.570356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.570359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.570745 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.571043 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.574343 (+ 3300us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.574598 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 02:30:20 dist-test-slave-pgkr krb5kdc[11685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156219, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:30:20.586094 11909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [term 1 FOLLOWER]: Refusing update from remote peer 2ec25333aa4c46c09005f82472b17cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:30:20.589033 11971 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:30:20.594698 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.568373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:40701 (local address 127.0.0.1:39698)
0808 02:30:20.568835 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:20.568858 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:20.568985 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:20.569979 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:20.569987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:20.570636 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:20.570945 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.570962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.573006 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.573014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.574983 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:20.574997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.576359 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.576365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:20.576664 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.577423 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:20.577446 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:20.580804 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:20.590650 (+ 9846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.590660 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.590667 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.591433 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.592100 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:20.592108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:20.592115 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:20.592344 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:20.593218 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:20.593226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:20.593449 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 02:30:20.593638 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 02:30:20.594098 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293}
I20260808 02:30:20.596379 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:20.569221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.19.231.60:40701)
0808 02:30:20.569428 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.569436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.569456 (+ 20us) server_negotiation.cc:407] Connection header received
0808 02:30:20.569517 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.569524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.569626 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.569739 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.571278 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.572797 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.575254 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:20.576184 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:20.576326 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:20.581277 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:20.581283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:20.581287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:20.581327 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:20.586007 (+ 4680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.591701 (+ 5694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.591709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.591715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.591921 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:20.592523 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:20.592531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:20.592537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:20.593032 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:20.595345 (+ 2313us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:20.595509 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 02:30:20.595855 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 02:30:20.607226 11857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063: Bootstrap starting.
I20260808 02:30:20.615725 11857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:20.618011 11857 log.cc:824] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:20.625489 11967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec25333aa4c46c09005f82472b17cf7. Latest consensus state: current_term: 1 leader_uuid: "2ec25333aa4c46c09005f82472b17cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.625854 11967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:20.629205 11857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063: No bootstrap required, opened a new log
I20260808 02:30:20.632477 11979 mvcc.cc:205] Tried to move back new op lower bound from 7316095879505838080 to 7316095879354077184. Current Snapshot: MvccSnapshot[applied={T|T < 7316095879505838080}]
W20260808 02:30:20.634881 11887 tablet.cc:2369] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:30:20.641726 11981 mvcc.cc:205] Tried to move back new op lower bound from 7316095879505838080 to 7316095879354077184. Current Snapshot: MvccSnapshot[applied={T|T < 7316095879505838080}]
I20260808 02:30:20.652176 11857 raft_consensus.cc:348] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
I20260808 02:30:20.653064 11857 raft_consensus.cc:729] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b446ac56fdc84bfcbae4ae08e6bc1063, State: Initialized, Role: FOLLOWER
I20260808 02:30:20.653929 11857 consensus_queue.cc:260] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } }
I20260808 02:30:20.658998 11991 sys_catalog.cc:455] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.661312 11945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ec25333aa4c46c09005f82472b17cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.662039 11945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:20.662799 11967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec25333aa4c46c09005f82472b17cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.663084 11967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:20.665325 11991 sys_catalog.cc:458] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:20.665136 11857 sys_catalog.cc:564] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:20.665591 11828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:20.666796 11829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ec25333aa4c46c09005f82472b17cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063"
I20260808 02:30:20.670482 11828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [term 1 FOLLOWER]: Refusing update from remote peer 2ec25333aa4c46c09005f82472b17cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:30:20.675873 11829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ec25333aa4c46c09005f82472b17cf7 in term 1.
I20260808 02:30:20.678380 11945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:30:20.688841 11976 catalog_manager.cc:1347] Generated new cluster ID: 4b38bac7827948c3be1394eb520bc0fb
I20260808 02:30:20.688987 11976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:20.689510 11865 master_runner.cc:186] Error getting master registration for 127.19.231.61:35953: OK, instance_id { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" instance_seqno: 1786156219832839 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:30:20.701365 11865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:30:20.705870 11991 sys_catalog.cc:455] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec25333aa4c46c09005f82472b17cf7. Latest consensus state: current_term: 1 leader_uuid: "2ec25333aa4c46c09005f82472b17cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.706211 11991 sys_catalog.cc:458] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:20.706827 12002 sys_catalog.cc:455] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ec25333aa4c46c09005f82472b17cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.708331 12002 sys_catalog.cc:458] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 02:30:20.714562 11994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:30:20.714779 11994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:30:20.736231 12008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ec25333aa4c46c09005f82472b17cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ec25333aa4c46c09005f82472b17cf7" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 37481 } } peers { permanent_uuid: "b446ac56fdc84bfcbae4ae08e6bc1063" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 35953 } } peers { permanent_uuid: "a9683409235042d1b4bac709d1bacedc" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 40701 } } }
I20260808 02:30:20.737272 12008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7 [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:20.756062 11976 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:20.758098 11976 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:20.775470 11976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ec25333aa4c46c09005f82472b17cf7: Generated new TSK 0
I20260808 02:30:20.777333 11976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:20.785883 12012 catalog_manager.cc:1259] Loaded cluster ID: 4b38bac7827948c3be1394eb520bc0fb
I20260808 02:30:20.786022 12012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063: loading cluster ID for follower catalog manager: success
I20260808 02:30:20.793857 12012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063: acquiring CA information for follower catalog manager: success
I20260808 02:30:20.796564 12012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b446ac56fdc84bfcbae4ae08e6bc1063: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:30:20.948637 11942 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:20.915405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.19.231.62:37481)
0808 02:30:20.915605 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.915611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.937732 (+ 22121us) server_negotiation.cc:407] Connection header received
0808 02:30:20.938276 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.938283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.938395 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.938479 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.948324 (+ 9845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52602: BlockingRecv error: recv got EOF from 127.0.0.1:52602 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 02:30:20.951676 11862 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:20.922249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.19.231.61:35953)
0808 02:30:20.922469 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.922475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.949656 (+ 27181us) server_negotiation.cc:407] Connection header received
0808 02:30:20.950158 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.950165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.950253 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.950333 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.951435 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44264: BlockingRecv error: recv got EOF from 127.0.0.1:44264 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 02:30:20.955178 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:20.952911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.19.231.60:40701)
0808 02:30:20.953113 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:20.953120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:20.953553 (+ 433us) server_negotiation.cc:407] Connection header received
0808 02:30:20.953801 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:20.953808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:20.953891 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:20.953964 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:20.954897 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39710: BlockingRecv error: recv got EOF from 127.0.0.1:39710 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:30:21.028846 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.009274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.19.231.61:35953)
0808 02:30:21.009567 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.009575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.020606 (+ 11031us) server_negotiation.cc:407] Connection header received
0808 02:30:21.021019 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.021025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.021277 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.021432 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.022243 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.023467 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.025558 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.026255 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.026350 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.027204 (+ 854us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.027948 (+ 744us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.028074 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.028343 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.028446 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260808 02:30:21.085239 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.067595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.19.231.60:40701)
0808 02:30:21.067821 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.067830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.083279 (+ 15449us) server_negotiation.cc:407] Connection header received
0808 02:30:21.083568 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.083575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.083725 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.083840 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.084974 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39716: BlockingRecv error: recv got EOF from 127.0.0.1:39716 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 02:30:21.141063 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.123251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.19.231.61:35953)
0808 02:30:21.123454 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.123461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.134298 (+ 10837us) server_negotiation.cc:407] Connection header received
0808 02:30:21.134600 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.134631 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.134736 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.134816 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.135525 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.136746 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.138820 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.139437 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.139542 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.140166 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.140326 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.140386 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.140624 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.140709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 02:30:21.162515 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.160368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.19.231.60:40701)
0808 02:30:21.160637 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.160644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.160831 (+ 187us) server_negotiation.cc:407] Connection header received
0808 02:30:21.161121 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.161127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.161230 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.161360 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.162252 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39720: BlockingRecv error: recv got EOF from 127.0.0.1:39720 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 02:30:21.179231 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.172032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.19.231.61:35953)
0808 02:30:21.172224 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.172230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.172765 (+ 535us) server_negotiation.cc:407] Connection header received
0808 02:30:21.172940 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.172947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.173068 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.173184 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.173656 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.174917 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.176933 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.177678 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.177910 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.178290 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.178448 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.178563 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.178783 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.178869 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 02:30:21.195174 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.192826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.19.231.60:40701)
0808 02:30:21.193002 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.193009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.193549 (+ 540us) server_negotiation.cc:407] Connection header received
0808 02:30:21.193741 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.193746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.193844 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.193964 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.194868 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39724: BlockingRecv error: recv got EOF from 127.0.0.1:39724 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 02:30:21.211930 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.204676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.19.231.61:35953)
0808 02:30:21.204911 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.204917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.205463 (+ 546us) server_negotiation.cc:407] Connection header received
0808 02:30:21.205670 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.205677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.205800 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.205912 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.206382 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.207675 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.209672 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.210397 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.210533 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.211065 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.211212 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.211314 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.211531 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.211619 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260808 02:30:21.229029 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.218990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.19.231.60:40701)
0808 02:30:21.219294 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.219300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.227290 (+ 7990us) server_negotiation.cc:407] Connection header received
0808 02:30:21.227502 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.227508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.227617 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.227740 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.228766 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39730: BlockingRecv error: recv got EOF from 127.0.0.1:39730 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
I20260808 02:30:21.247680 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.240238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.19.231.61:35953)
0808 02:30:21.240433 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.240440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.240953 (+ 513us) server_negotiation.cc:407] Connection header received
0808 02:30:21.241197 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.241203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.241288 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.241361 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.241981 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.243182 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.245180 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.245897 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.246017 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.246770 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.246941 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.247005 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.247260 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.247349 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 02:30:21.258128 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.255701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.19.231.60:40701)
0808 02:30:21.255944 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.255955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.256493 (+ 538us) server_negotiation.cc:407] Connection header received
0808 02:30:21.256681 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.256688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.256791 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.256909 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.257884 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39742: BlockingRecv error: recv got EOF from 127.0.0.1:39742 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:30:21.285807 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.277959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.19.231.61:35953)
0808 02:30:21.278176 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.278182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.279264 (+ 1082us) server_negotiation.cc:407] Connection header received
0808 02:30:21.279460 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.279465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.279576 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.279697 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.280301 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.281467 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.283603 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.284269 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.284420 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.284979 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.285150 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.285236 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.285409 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.285512 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 02:30:21.310635 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.308281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.19.231.60:40701)
0808 02:30:21.308536 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.308546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.309000 (+ 454us) server_negotiation.cc:407] Connection header received
0808 02:30:21.309250 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.309254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.309327 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.309414 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.310326 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39750: BlockingRecv error: recv got EOF from 127.0.0.1:39750 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:30:21.352872 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.346599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.19.231.61:35953)
0808 02:30:21.346922 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.346928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.347236 (+ 308us) server_negotiation.cc:407] Connection header received
0808 02:30:21.347459 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.347467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.347538 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.347617 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.348107 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.348849 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.350886 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.351446 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.351553 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.352075 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.352180 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.352268 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.352450 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.352516 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260808 02:30:21.362244 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.359928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.19.231.60:40701)
0808 02:30:21.360143 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.360151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.360745 (+ 594us) server_negotiation.cc:407] Connection header received
0808 02:30:21.360900 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.360904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.360988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.361067 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.362058 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39762: BlockingRecv error: recv got EOF from 127.0.0.1:39762 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:30:21.435490 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.430171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.19.231.61:35953)
0808 02:30:21.430422 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.430428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.430755 (+ 327us) server_negotiation.cc:407] Connection header received
0808 02:30:21.430997 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.431001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.431070 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.431169 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.431620 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.432395 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.433817 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.434250 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.434388 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.434792 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.434892 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.434987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.435137 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.435219 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260808 02:30:21.442536 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.440783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.19.231.60:40701)
0808 02:30:21.441022 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.441028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.441262 (+ 234us) server_negotiation.cc:407] Connection header received
0808 02:30:21.441418 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.441422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.441504 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.441610 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.442271 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39774: BlockingRecv error: recv got EOF from 127.0.0.1:39774 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 02:30:21.580404 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.574383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.19.231.61:35953)
0808 02:30:21.574680 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.574688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.574978 (+ 290us) server_negotiation.cc:407] Connection header received
0808 02:30:21.575263 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.575268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.575359 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.575471 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.576015 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.576893 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.578444 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.578970 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.579120 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.579604 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.579720 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.579828 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.579991 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.580084 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 02:30:21.588352 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:21.586432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.19.231.60:40701)
0808 02:30:21.586766 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.586771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.587038 (+ 267us) server_negotiation.cc:407] Connection header received
0808 02:30:21.587184 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.587190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.587270 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.587367 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:21.588155 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39784: BlockingRecv error: recv got EOF from 127.0.0.1:39784 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
I20260808 02:30:21.716625 11994 catalog_manager.cc:1259] Loaded cluster ID: 4b38bac7827948c3be1394eb520bc0fb
I20260808 02:30:21.716754 11994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: loading cluster ID for follower catalog manager: success
I20260808 02:30:21.719309 11994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: acquiring CA information for follower catalog manager: success
I20260808 02:30:21.721053 11994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9683409235042d1b4bac709d1bacedc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:30:21.852852 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.847929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.19.231.61:35953)
0808 02:30:21.848136 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.848142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.848335 (+ 193us) server_negotiation.cc:407] Connection header received
0808 02:30:21.848462 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.848467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.848587 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.848683 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.849091 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.849866 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.851181 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.851617 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.851729 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.852100 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.852214 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.852291 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.852471 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.852600 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 02:30:21.864437 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.858908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.19.231.60:40701)
0808 02:30:21.859132 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.859137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.859283 (+ 146us) server_negotiation.cc:407] Connection header received
0808 02:30:21.859420 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.859424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.859612 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.859709 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.860072 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.860880 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.862203 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.862664 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.862792 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.863288 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.863853 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.863960 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.864050 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.864116 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:30:21.878252 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.870448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.19.231.61:35953)
0808 02:30:21.870715 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.870727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.871563 (+ 836us) server_negotiation.cc:407] Connection header received
0808 02:30:21.871816 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.871823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.871941 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.872044 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.872665 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.873671 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.875866 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.876592 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.876758 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.877309 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.877461 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.877580 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.877738 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.877851 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:30:21.880613 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.870436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.19.231.62:37481)
0808 02:30:21.872344 (+ 1908us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.872356 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.873419 (+ 1063us) server_negotiation.cc:407] Connection header received
0808 02:30:21.873584 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.873589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.873817 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.874074 (+ 257us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.874749 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.875723 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.878219 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.878872 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.879011 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.879578 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.879978 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.880112 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.880222 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.880307 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":1536,"threads_started":1}
I20260808 02:30:21.881480 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:21.873143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.19.231.60:40701)
0808 02:30:21.873363 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:21.873372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:21.873764 (+ 392us) server_negotiation.cc:407] Connection header received
0808 02:30:21.874013 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:21.874024 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:21.874147 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:21.874258 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:21.875167 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.876099 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.878926 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:21.879707 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:21.879896 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:21.880635 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:21.880780 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:21.880882 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:21.881141 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 02:30:21.881235 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 02:30:21.886790 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11701
I20260808 02:30:21.887821 11768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:22.002949 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11701
W20260808 02:30:22.029274 11801 connection.cc:535] server connection from 127.0.0.1:44210 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 02:30:22.029706 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11781
I20260808 02:30:22.030864 11852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:22.141676 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11781
I20260808 02:30:22.168259 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 11865
I20260808 02:30:22.169502 11929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:22.279422 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 11865
2026-08-08T02:30:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5082 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 02:30:22 dist-test-slave-pgkr krb5kdc[12148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 02:30:22 dist-test-slave-pgkr krb5kdc[12148](info): set up 2 sockets
Aug 08 02:30:22 dist-test-slave-pgkr krb5kdc[12148](info): commencing operation
krb5kdc: starting...
W20260808 02:30:23.478741 20380 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.006s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/test-user.keytab.
Aug 08 02:30:23 dist-test-slave-pgkr krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T02:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T02:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.231.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
Entry for principal kudu/127.19.231.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab.
I20260808 02:30:23.853663 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:34189
--webserver_interface=127.19.231.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:34189,127.19.231.61:46287,127.19.231.60:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 02:30:24.228667 12163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:30:24.229110 12163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:24.229242 12163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:24.229393 12163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:24.237668 12163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:24.237808 12163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:24.237936 12163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:24.238020 12163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:24.238111 12163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:24.250716 12163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:34189,127.19.231.61:46287,127.19.231.60:43725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.62.keytab
--principal=kudu/127.19.231.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.62:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.231.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:24.252599 12163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:24.254329 12163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:24.269464 12171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:24.270318 12170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:24.270805 12174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:24.271164 12163 server_base.cc:1034] running on GCE node
I20260808 02:30:24.271999 12163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:24.273705 12163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:24.274971 12163 hybrid_clock.cc:648] HybridClock initialized: now 1786156224274866 us; error 69 us; skew 500 ppm
I20260808 02:30:24.275436 12163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:24 dist-test-slave-pgkr krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:24.279807 12163 init.cc:377] Logged in from keytab as kudu/127.19.231.62@KRBTEST.COM (short username kudu)
I20260808 02:30:24.281749 12163 webserver.cc:460] Webserver started at http://127.19.231.62:32947/ using document root <none> and password file <none>
I20260808 02:30:24.282773 12163 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:24.282904 12163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:24.283314 12163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:24.287354 12163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data/instance:
uuid: "cca764302396443fb616ef6e1c1bd5aa"
format_stamp: "Formatted at 2026-08-08 02:30:24 on dist-test-slave-pgkr"
server_key: "06157519f4f086becaa397157929b477"
server_key_iv: "917dded0f7a0bcba674c7d1baa23931d"
server_key_version: "encryptionkey@0"
I20260808 02:30:24.288458 12163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/wal/instance:
uuid: "cca764302396443fb616ef6e1c1bd5aa"
format_stamp: "Formatted at 2026-08-08 02:30:24 on dist-test-slave-pgkr"
server_key: "06157519f4f086becaa397157929b477"
server_key_iv: "917dded0f7a0bcba674c7d1baa23931d"
server_key_version: "encryptionkey@0"
I20260808 02:30:24.296044 12163 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260808 02:30:24.300693 12182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:24.302563 12163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 02:30:24.302786 12163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/wal
uuid: "cca764302396443fb616ef6e1c1bd5aa"
format_stamp: "Formatted at 2026-08-08 02:30:24 on dist-test-slave-pgkr"
server_key: "06157519f4f086becaa397157929b477"
server_key_iv: "917dded0f7a0bcba674c7d1baa23931d"
server_key_version: "encryptionkey@0"
I20260808 02:30:24.303058 12163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:24.328615 12163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:24.332795 12163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:24.333055 12163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:24.352654 12163 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.62:34189
I20260808 02:30:24.352674 12238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.62:34189 every 8 connection(s)
I20260808 02:30:24.354067 12163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/data/info.pb
I20260808 02:30:24.357976 12239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:24.361325 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 12163
I20260808 02:30:24.361527 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-0/wal/instance
I20260808 02:30:24.362098 20380 external_mini_cluster.cc:1428] Setting key 2c3f5f33dedaac94e089bd3f53039e5d
I20260808 02:30:24.366935 12239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:24 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:24.393710 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.370946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34189 (local address 127.0.0.1:37184)
0808 02:30:24.371872 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.371933 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:24.374868 (+ 2935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:24.375806 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:24.375816 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:24.376701 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:24.377248 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:24.377277 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.378952 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.378959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:24.380071 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:24.380083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.381415 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.381420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:24.381667 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:24.382373 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:24.382396 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:24.385089 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:24.389637 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:24.389647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:24.389656 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:24.390053 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:24.390532 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:24.390535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:24.390538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:24.390746 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:24.391757 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:24.391763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:24.391958 (+ 195us) client_negotiation.cc:769] Sending connection context
0808 02:30:24.392171 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 02:30:24.392585 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":57,"thread_start_us":369,"threads_started":1}
I20260808 02:30:24.393750 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.372213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.19.231.62:34189)
0808 02:30:24.375123 (+ 2910us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:24.375129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:24.375171 (+ 42us) server_negotiation.cc:407] Connection header received
0808 02:30:24.375313 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:24.375318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:24.375562 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:24.375724 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:24.377490 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.378789 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.380217 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.381192 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.381403 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:24.385340 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:24.385353 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:24.385372 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:24.385416 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:24.389497 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:24.390190 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:24.390200 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:24.390205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:24.390406 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:24.390846 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:24.390853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:24.390877 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:24.391602 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:24.391882 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:24.392378 (+ 496us) server_negotiation.cc:299] Negotiation successful
0808 02:30:24.392746 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":2742,"thread_start_us":2592,"threads_started":1}
I20260808 02:30:24.397203 12239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:24.398561 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.398022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57228)
0808 02:30:24.398260 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.398404 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260808 02:30:24.398937 12186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
I20260808 02:30:24.400667 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.400204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57242)
0808 02:30:24.400396 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.400486 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 02:30:24.400924 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:46287: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.231.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab.
Entry for principal kudu/127.19.231.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab.
I20260808 02:30:24.445292 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:46287
--webserver_interface=127.19.231.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab
--principal=kudu/127.19.231.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:34189,127.19.231.61:46287,127.19.231.60:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:30:24.450340 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } attempt: 1
I20260808 02:30:24.451617 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.451176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57244)
0808 02:30:24.451374 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.451479 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 02:30:24.452591 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.452228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57254)
0808 02:30:24.452391 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.452466 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260808 02:30:24.452757 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:46287: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
I20260808 02:30:24.521050 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } attempt: 2
I20260808 02:30:24.522385 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.521802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57256)
0808 02:30:24.522085 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.522207 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 02:30:24.523571 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.523118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57258)
0808 02:30:24.523273 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.523378 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 02:30:24.523892 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:46287: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
I20260808 02:30:24.615176 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } attempt: 3
I20260808 02:30:24.616384 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.615943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57264)
0808 02:30:24.616145 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.616246 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 02:30:24.617533 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.617159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57280)
0808 02:30:24.617308 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.617388 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:30:24.617799 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:46287: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
I20260808 02:30:24.761088 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } attempt: 4
I20260808 02:30:24.762522 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.761856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57286)
0808 02:30:24.762271 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.762363 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
I20260808 02:30:24.763854 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.763454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57300)
0808 02:30:24.763616 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.763716 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260808 02:30:24.764196 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.61:46287: Network error: Client connection negotiation failed: client connection to 127.19.231.61:46287: connect: Connection refused (error 111)
W20260808 02:30:24.844461 12246 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:30:24.844851 12246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:24.844959 12246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:24.845132 12246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:24.852622 12246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:24.852720 12246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:24.852856 12246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:24.852950 12246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:24.853062 12246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:24.863499 12246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:34189,127.19.231.61:46287,127.19.231.60:43725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.61.keytab
--principal=kudu/127.19.231.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.61:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.231.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:24.864950 12246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:24.866376 12246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:24.879024 12251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:24.878971 12254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:24.879082 12252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:24.879882 12246 server_base.cc:1034] running on GCE node
I20260808 02:30:24.880722 12246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:24.882189 12246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:24.883469 12246 hybrid_clock.cc:648] HybridClock initialized: now 1786156224883402 us; error 34 us; skew 500 ppm
I20260808 02:30:24.883844 12246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:24 dist-test-slave-pgkr krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:24.887404 12246 init.cc:377] Logged in from keytab as kudu/127.19.231.61@KRBTEST.COM (short username kudu)
I20260808 02:30:24.888986 12246 webserver.cc:460] Webserver started at http://127.19.231.61:46725/ using document root <none> and password file <none>
I20260808 02:30:24.889757 12246 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:24.889897 12246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:24.890264 12246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:24.893620 12246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data/instance:
uuid: "20762a3ea21a4705b9890f034f5592ba"
format_stamp: "Formatted at 2026-08-08 02:30:24 on dist-test-slave-pgkr"
server_key: "ba2fcee683ddb4bf70791d7a94124fc7"
server_key_iv: "547c6c61a7ceef9efa1e18a6c19f5303"
server_key_version: "encryptionkey@0"
I20260808 02:30:24.894488 12246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/wal/instance:
uuid: "20762a3ea21a4705b9890f034f5592ba"
format_stamp: "Formatted at 2026-08-08 02:30:24 on dist-test-slave-pgkr"
server_key: "ba2fcee683ddb4bf70791d7a94124fc7"
server_key_iv: "547c6c61a7ceef9efa1e18a6c19f5303"
server_key_version: "encryptionkey@0"
I20260808 02:30:24.900254 12246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 02:30:24.904006 12261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:24.905371 12246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:30:24.905612 12246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/wal
uuid: "20762a3ea21a4705b9890f034f5592ba"
format_stamp: "Formatted at 2026-08-08 02:30:24 on dist-test-slave-pgkr"
server_key: "ba2fcee683ddb4bf70791d7a94124fc7"
server_key_iv: "547c6c61a7ceef9efa1e18a6c19f5303"
server_key_version: "encryptionkey@0"
I20260808 02:30:24.905858 12246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:24.919579 12246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:24.923719 12246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:24.924021 12246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:24.944160 12246 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.61:46287
I20260808 02:30:24.944205 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.61:46287 every 8 connection(s)
I20260808 02:30:24.945755 12246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/data/info.pb
I20260808 02:30:24.946450 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 12246
I20260808 02:30:24.946679 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-1/wal/instance
I20260808 02:30:24.947325 20380 external_mini_cluster.cc:1428] Setting key 9005e4cca9f79e955a533750be3865ed
I20260808 02:30:24.949970 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:24.957137 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:24 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:24.979259 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.959931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.19.231.62:34189)
0808 02:30:24.960510 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:24.960516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:24.961405 (+ 889us) server_negotiation.cc:407] Connection header received
0808 02:30:24.963422 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:24.963427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:24.963524 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:24.963641 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:24.965172 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.965924 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.967471 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.967904 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.968036 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:24.971811 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:24.971820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:24.971828 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:24.971903 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:24.975804 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:24.976556 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:24.976564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:24.976570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:24.976737 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:24.977174 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:24.977182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:24.977188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:24.977605 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:24.977793 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:24.978444 (+ 651us) server_negotiation.cc:299] Negotiation successful
0808 02:30:24.978761 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":349,"threads_started":1}
I20260808 02:30:24.979938 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.960459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34189 (local address 127.0.0.1:37196)
0808 02:30:24.961231 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.961275 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:24.963155 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:24.963883 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:24.963891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:24.964413 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:24.964950 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:24.964966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.966088 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.966092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:24.967326 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:24.967338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.968155 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.968163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:24.968379 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:24.969070 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:24.969097 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:24.971547 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:24.975952 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:24.975959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:24.975971 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:24.976417 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:24.976851 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:24.976854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:24.976857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:24.977025 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:24.977726 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:24.977732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:24.978036 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 02:30:24.978287 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 02:30:24.978745 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":329,"threads_started":1}
I20260808 02:30:24.981945 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.231.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.231.60@KRBTEST.COM" created.
Aug 08 02:30:24 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:30:24.998308 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.982867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57306)
0808 02:30:24.983099 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:24.983124 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:24.983240 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:24.984263 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:24.984270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:24.984683 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:24.984830 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:24.984838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.986314 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.986319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:24.987792 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:24.987811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.988640 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.988649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:24.988880 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:24.989505 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:24.989532 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:24.992062 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:24.995777 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:24.995784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:24.995791 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:24.996151 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:24.996523 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:24.996528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:24.996532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:24.996678 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:24.997295 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:24.997301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:24.997462 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 02:30:24.997590 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:30:24.997892 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 02:30:24.998368 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:24.983014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.19.231.61:46287)
0808 02:30:24.983783 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:24.983789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:24.983815 (+ 26us) server_negotiation.cc:407] Connection header received
0808 02:30:24.983858 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:24.983864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:24.984035 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:24.984177 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:24.984989 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.986179 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.987935 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:24.988481 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:24.988580 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:24.992289 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:24.992301 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:24.992315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:24.992344 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:24.995654 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:24.996288 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:24.996296 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:24.996300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:24.996419 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:24.996803 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:24.996808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:24.996811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:24.997164 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:24.997390 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:24.997777 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 02:30:24.997993 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":450,"threads_started":1}
I20260808 02:30:25.001073 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:25.002327 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.001868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37098)
0808 02:30:25.002040 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.002159 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 02:30:25.002668 12263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.004559 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.004109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37114)
0808 02:30:25.004309 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.004411 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 02:30:25.004812 12316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.231.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab.
Entry for principal kudu/127.19.231.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab.
I20260808 02:30:25.028193 20380 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
/tmp/dist-test-taskOT6xt1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:43725
--webserver_interface=127.19.231.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab
--principal=kudu/127.19.231.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.231.20:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.231.62:34189,127.19.231.61:46287,127.19.231.60:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 02:30:25.054172 12316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 1
I20260808 02:30:25.055645 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.054978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37130)
0808 02:30:25.055291 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.055429 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 02:30:25.056977 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.056563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37138)
0808 02:30:25.056767 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.056857 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 02:30:25.057281 12316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.063462 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } attempt: 5
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:30:25.077226 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.064183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:57318)
0808 02:30:25.064417 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.064437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:25.064622 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:25.064991 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:25.064996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:25.065430 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:25.065609 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.065617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.066589 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.066597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.067516 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.067529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.068304 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.068310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.068529 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.069148 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:25.069166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:25.071288 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:25.074737 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.074743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.074747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.075131 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.075548 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.075553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.075557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.075702 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.076249 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:25.076254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:25.076405 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 02:30:25.076545 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 02:30:25.076860 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 02:30:25.077217 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.064236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.19.231.61:46287)
0808 02:30:25.064430 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.064437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.064519 (+ 82us) server_negotiation.cc:407] Connection header received
0808 02:30:25.064771 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.064775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.064854 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.064955 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.065751 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.066457 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.067686 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.068158 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.068292 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.071485 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.071496 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.071500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.071530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.074561 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.075282 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.075288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.075292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.075430 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.075820 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.075826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.075829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.076122 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.076271 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.076656 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.076881 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 02:30:25.078550 12239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:25.079833 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.079394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37152)
0808 02:30:25.079559 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.079666 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 02:30:25.080926 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.080559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37168)
0808 02:30:25.080735 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.080820 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 02:30:25.081141 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.125540 12316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 2
I20260808 02:30:25.126852 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.126333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37174)
0808 02:30:25.126533 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.126691 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 02:30:25.127827 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.127494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37180)
0808 02:30:25.127635 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.127708 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 02:30:25.128057 12316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.132419 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 1
I20260808 02:30:25.133538 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.133091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37184)
0808 02:30:25.133278 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.133369 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 02:30:25.134691 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.134243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37194)
0808 02:30:25.134396 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.134477 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 02:30:25.134908 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.203176 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 2
I20260808 02:30:25.204612 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.204024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37196)
0808 02:30:25.204277 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.204393 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 02:30:25.205838 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.205443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37206)
0808 02:30:25.205603 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.205704 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 02:30:25.206120 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.219291 12316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 3
I20260808 02:30:25.220522 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.220001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37222)
0808 02:30:25.220248 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.220353 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 02:30:25.221661 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.221212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:37234)
0808 02:30:25.221462 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.221545 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260808 02:30:25.221941 12316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.312392 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 3
I20260808 02:30:25.313669 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.313115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:41942)
0808 02:30:25.313332 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.313434 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 02:30:25.314921 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.314415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:41948)
0808 02:30:25.314719 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.314806 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260808 02:30:25.315153 12239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
I20260808 02:30:25.365193 12316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 4
I20260808 02:30:25.366384 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.365917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:41964)
0808 02:30:25.366144 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.366233 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 02:30:25.367353 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.367045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:41972)
0808 02:30:25.367185 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.367253 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 02:30:25.367570 12316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.231.60:43725: Network error: Client connection negotiation failed: client connection to 127.19.231.60:43725: connect: Connection refused (error 111)
W20260808 02:30:25.388821 12324 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 02:30:25.389186 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 02:30:25.389312 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 02:30:25.389448 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 02:30:25.396931 12324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 02:30:25.397047 12324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 02:30:25.397173 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 02:30:25.397259 12324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 02:30:25.397367 12324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 02:30:25.407833 12324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.231.20:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.231.62:34189,127.19.231.61:46287,127.19.231.60:43725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/krb5kdc/kudu_127.19.231.60.keytab
--principal=kudu/127.19.231.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.231.60:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.231.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 01:07:42 UTC on ccycloud.quasar-vfhkhu.root.comops.site
build id 4573758
ASAN enabled
I20260808 02:30:25.409332 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 02:30:25.410768 12324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 02:30:25.422601 12329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:25.422675 12331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 02:30:25.422806 12333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 02:30:25.424232 12324 server_base.cc:1034] running on GCE node
I20260808 02:30:25.425125 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 02:30:25.426779 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 02:30:25.427999 12324 hybrid_clock.cc:648] HybridClock initialized: now 1786156225427931 us; error 47 us; skew 500 ppm
I20260808 02:30:25.428362 12324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156225, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 02:30:25.431689 12324 init.cc:377] Logged in from keytab as kudu/127.19.231.60@KRBTEST.COM (short username kudu)
I20260808 02:30:25.433305 12324 webserver.cc:460] Webserver started at http://127.19.231.60:40931/ using document root <none> and password file <none>
I20260808 02:30:25.434036 12324 fs_manager.cc:359] Metadata directory not provided
I20260808 02:30:25.434165 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 02:30:25.434500 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 02:30:25.437674 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data/instance:
uuid: "3232f92d73924500939f3c8a5ea9cb9c"
format_stamp: "Formatted at 2026-08-08 02:30:25 on dist-test-slave-pgkr"
server_key: "5ab8924ff5e6944e0a9dd0162d31519a"
server_key_iv: "fd50252fe84a99238a3b3506b68c9096"
server_key_version: "encryptionkey@0"
I20260808 02:30:25.438524 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/wal/instance:
uuid: "3232f92d73924500939f3c8a5ea9cb9c"
format_stamp: "Formatted at 2026-08-08 02:30:25 on dist-test-slave-pgkr"
server_key: "5ab8924ff5e6944e0a9dd0162d31519a"
server_key_iv: "fd50252fe84a99238a3b3506b68c9096"
server_key_version: "encryptionkey@0"
I20260808 02:30:25.443928 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 02:30:25.447553 12340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 02:30:25.448827 12324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 02:30:25.449009 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data,/tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/wal
uuid: "3232f92d73924500939f3c8a5ea9cb9c"
format_stamp: "Formatted at 2026-08-08 02:30:25 on dist-test-slave-pgkr"
server_key: "5ab8924ff5e6944e0a9dd0162d31519a"
server_key_iv: "fd50252fe84a99238a3b3506b68c9096"
server_key_version: "encryptionkey@0"
I20260808 02:30:25.449226 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 02:30:25.461021 12324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 02:30:25.465283 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 02:30:25.465546 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 02:30:25.484388 12324 rpc_server.cc:307] RPC server started. Bound to: 127.19.231.60:43725
I20260808 02:30:25.484396 12397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.231.60:43725 every 8 connection(s)
I20260808 02:30:25.485777 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/data/info.pb
I20260808 02:30:25.487171 20380 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu as pid 12324
I20260808 02:30:25.487380 20380 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOT6xt1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786155839876607-20380-0/minicluster-data/master-2/wal/instance
I20260808 02:30:25.488015 20380 external_mini_cluster.cc:1428] Setting key 7092b865dfccbe6420b7fa3c071b7bb0
I20260808 02:30:25.490512 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 02:30:25.492373 12239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 4
I20260808 02:30:25.499293 12398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.62@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:30:25.514087 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.493041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:41978)
0808 02:30:25.493251 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.493271 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:25.493390 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:25.494216 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:25.494223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:25.494830 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:25.495052 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.495066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.497506 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.497513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.498949 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.498967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.500226 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.500233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.500422 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.501112 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:25.501139 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:25.504151 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:25.510210 (+ 6059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.510213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.510216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.510950 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.511506 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.511511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.511515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.511693 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.512796 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:25.512803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:25.512958 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 02:30:25.513086 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 02:30:25.513392 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 02:30:25.516109 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.493221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.19.231.60:43725)
0808 02:30:25.493845 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.493849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.493867 (+ 18us) server_negotiation.cc:407] Connection header received
0808 02:30:25.493929 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.493933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.493999 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.495223 (+ 1224us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.495562 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.497284 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.499106 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.500024 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.500755 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.504573 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.504587 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.504611 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.504669 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.510067 (+ 5398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.511110 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.511118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.511128 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.511373 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.511824 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.511832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.511838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.512615 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.514387 (+ 1772us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.514555 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.514903 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":514,"spinlock_wait_cycles":22272,"thread_start_us":362,"threads_started":1}
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156225, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:25.516939 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.489875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.19.231.60:43725)
0808 02:30:25.490720 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.490737 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.490778 (+ 41us) server_negotiation.cc:407] Connection header received
0808 02:30:25.492420 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.492440 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.493542 (+ 1102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.494015 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.495864 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.497350 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.499273 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.500017 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.500243 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.506705 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.506716 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.506721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.506765 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.512058 (+ 5293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.512797 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.512807 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.512810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.512925 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.513312 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.513320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.513323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.513572 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.513889 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.514308 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.516467 (+ 2159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":577,"thread_start_us":337,"threads_started":1}
I20260808 02:30:25.524871 12239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa: Bootstrap starting.
I20260808 02:30:25.524950 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.502503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.62:34189 (local address 127.0.0.1:45942)
0808 02:30:25.506501 (+ 3998us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.506847 (+ 346us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:25.507073 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:25.507496 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:25.507505 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:25.508428 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:25.508931 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.508946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.510486 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.510491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.512102 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.512113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.513303 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.513313 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.513517 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.514468 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:25.514490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:25.517656 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:25.522145 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.522157 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.522164 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.522716 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.523113 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.523120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.523125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.523304 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.524000 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:25.524009 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:25.524201 (+ 192us) client_negotiation.cc:769] Sending connection context
0808 02:30:25.524390 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 02:30:25.524676 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3697,"thread_start_us":3514,"threads_started":1}
I20260808 02:30:25.525764 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.502484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.19.231.62:34189)
0808 02:30:25.506862 (+ 4378us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.506871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.506968 (+ 97us) server_negotiation.cc:407] Connection header received
0808 02:30:25.507213 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.507219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.507322 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.507448 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.509096 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.510274 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.512282 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.512958 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.513082 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.517843 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.517848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.517851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.517881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.521995 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.522849 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.522855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.522859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.523006 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.523442 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.523448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.523453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.523846 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.523995 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.525285 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.525466 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4223,"thread_start_us":3253,"threads_started":1}
I20260808 02:30:25.527621 12398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.62@KRBTEST.COM
I20260808 02:30:25.530397 12239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:25.531973 12239 log.cc:824] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa: Log is configured to *not* fsync() on all Append() calls
I20260808 02:30:25.535691 12239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa: No bootstrap required, opened a new log
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156225, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
I20260808 02:30:25.537408 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.519373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.19.231.62:34189)
0808 02:30:25.520030 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.520037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.520060 (+ 23us) server_negotiation.cc:407] Connection header received
0808 02:30:25.520232 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.520237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.520303 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.520408 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.521675 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.522913 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.524433 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.525124 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.525263 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.530803 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.530811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.530816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.530862 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.534339 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.535177 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.535182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.535185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.535358 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.535798 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.535803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.535807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.536102 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.536368 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.536662 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.536895 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":312,"threads_started":1}
I20260808 02:30:25.544767 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.528456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.19.231.61:46287)
0808 02:30:25.528664 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.528672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.528703 (+ 31us) server_negotiation.cc:407] Connection header received
0808 02:30:25.528872 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.528879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.528968 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.529066 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.529954 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.530960 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.532358 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.533270 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.533405 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.537200 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.537208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.537214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.537256 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.541319 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.542320 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.542326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.542330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.542517 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.543040 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.543046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.543050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.543403 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.543574 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.544109 (+ 535us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.544335 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 02:30:25.544900 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.528454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.61:46287 (local address 127.0.0.1:37962)
0808 02:30:25.528641 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.528659 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:25.528746 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:25.529104 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:25.529109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:25.529594 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:25.529788 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.529800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.531104 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.531109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.532213 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.532226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.533634 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.533642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.533783 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.534307 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:25.534324 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:25.536969 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:25.541480 (+ 4511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.541488 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.541495 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.542175 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.542672 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.542680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.542686 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.542903 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.543551 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:25.543559 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:25.543784 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 02:30:25.543997 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 02:30:25.544384 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 02:30:25.546209 12398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 02:30:25.546725 12239 raft_consensus.cc:348] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.547150 12239 raft_consensus.cc:729] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca764302396443fb616ef6e1c1bd5aa, State: Initialized, Role: FOLLOWER
I20260808 02:30:25.547755 12239 consensus_queue.cc:260] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.551196 12409 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.551735 12409 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:25.552686 12239 sys_catalog.cc:564] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.231.61@KRBTEST.COM
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156225, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.60@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:30:25.562211 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.539188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.19.231.61:46287)
0808 02:30:25.545371 (+ 6183us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.545377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.545396 (+ 19us) server_negotiation.cc:407] Connection header received
0808 02:30:25.545445 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.545450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.545530 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.545688 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.547655 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.548908 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.550148 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.550573 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.551423 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.555751 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.555756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.555759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.555788 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.559238 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.560097 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.560104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.560109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.560265 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.560649 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.560654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.560658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.560928 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.561188 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.561507 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.561733 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6084,"thread_start_us":9307,"threads_started":1}
I20260808 02:30:25.565340 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.546952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:41990)
0808 02:30:25.547416 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.547436 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:25.547563 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:25.548025 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:25.548031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:25.548545 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:25.548806 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.548825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.550562 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.550570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.552320 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.552334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.553470 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.553475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.553673 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.554428 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:25.554461 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:25.557618 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:25.562250 (+ 4632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.562258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.562264 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.562800 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.563260 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.563268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.563275 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.563487 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.564299 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:25.564305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:25.564461 (+ 156us) client_negotiation.cc:769] Sending connection context
0808 02:30:25.564636 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 02:30:25.564971 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260808 02:30:25.565789 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.546983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.19.231.60:43725)
0808 02:30:25.547169 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.547175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.547491 (+ 316us) server_negotiation.cc:407] Connection header received
0808 02:30:25.547675 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.547680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.547764 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.547836 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.548995 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.550368 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.552463 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.553082 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.553184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.557857 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.557867 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.557875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.557923 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.562079 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.562934 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.562939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.562943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.563135 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.563644 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.563652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.563658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.564173 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.564360 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.564810 (+ 450us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.565186 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":64}
I20260808 02:30:25.571002 12398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: Bootstrap starting.
I20260808 02:30:25.572264 20380 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 02:30:25.574760 12211 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cca764302396443fb616ef6e1c1bd5aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45954
I20260808 02:30:25.575217 12398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:25.575377 12211 raft_consensus.cc:482] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 02:30:25.575639 12211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:25.576622 12398 log.cc:824] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: Log is configured to *not* fsync() on all Append() calls
W20260808 02:30:25.577504 12420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:30:25.577688 12420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:30:25.579002 12211 raft_consensus.cc:504] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.580453 12211 leader_election.cc:288] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [CANDIDATE]: Term 1 election: Requested vote from peers 20762a3ea21a4705b9890f034f5592ba (127.19.231.61:46287), 3232f92d73924500939f3c8a5ea9cb9c (127.19.231.60:43725)
I20260808 02:30:25.572448 12163 master_runner.cc:186] Error getting master registration for 127.19.231.61:46287: OK, instance_id { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" instance_seqno: 1786156224938459 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:30:25.586225 12398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: No bootstrap required, opened a new log
W20260808 02:30:25.585852 12188 tablet.cc:2369] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:30:25.597398 12163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:30:25.597759 12398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.598452 12398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3232f92d73924500939f3c8a5ea9cb9c, State: Initialized, Role: FOLLOWER
I20260808 02:30:25.599114 12398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.606144 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.608587 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:25.608445 12398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:25.608692 12372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cca764302396443fb616ef6e1c1bd5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3232f92d73924500939f3c8a5ea9cb9c"
I20260808 02:30:25.609443 12372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:25.613483 12372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca764302396443fb616ef6e1c1bd5aa in term 1.
I20260808 02:30:25.614246 12184 leader_election.cc:302] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3232f92d73924500939f3c8a5ea9cb9c, cca764302396443fb616ef6e1c1bd5aa; no voters:
I20260808 02:30:25.615254 12409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 02:30:25.624091 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.587165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.19.231.62:34189)
0808 02:30:25.587391 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.587401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.596134 (+ 8733us) server_negotiation.cc:407] Connection header received
0808 02:30:25.596446 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.596452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.596539 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.596614 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.599694 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.601150 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.609923 (+ 8773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.610806 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.610963 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.615317 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.615327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.615339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.615397 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.619840 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.620951 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.620960 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.620967 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.621157 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.621712 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.621720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.621726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.622255 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.622552 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.623336 (+ 784us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.623609 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 02:30:25.631383 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.586745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.19.231.60:43725)
0808 02:30:25.586949 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.586956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.600413 (+ 13457us) server_negotiation.cc:407] Connection header received
0808 02:30:25.600983 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.600991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.601224 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.601332 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.604168 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.605481 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.607710 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.608443 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.608537 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.613054 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.613061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.613065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.613105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.618782 (+ 5677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.626914 (+ 8132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.626920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.626924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.627134 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.628172 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.628181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.628184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.628697 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.629295 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.630544 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.630856 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 02:30:25.639889 12409 raft_consensus.cc:686] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [term 1 LEADER]: Becoming Leader. State: Replica: cca764302396443fb616ef6e1c1bd5aa, State: Running, Role: LEADER
I20260808 02:30:25.640777 12409 consensus_queue.cc:237] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.644726 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.595371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.19.231.61:46287)
0808 02:30:25.595607 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.595616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.607694 (+ 12078us) server_negotiation.cc:407] Connection header received
0808 02:30:25.608181 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.608190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.608288 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.608362 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.620317 (+ 11955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.621914 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.626245 (+ 4331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.627130 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.627257 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.633499 (+ 6242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.633506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.633520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.633571 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.637862 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.639765 (+ 1903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.639771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.639775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.640142 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.641086 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.641094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.641099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.641791 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.642202 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.643350 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.643807 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:30:25.649564 12432 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader cca764302396443fb616ef6e1c1bd5aa. Latest consensus state: current_term: 1 leader_uuid: "cca764302396443fb616ef6e1c1bd5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.650054 12432 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:25.654203 12441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 02:30:25.656651 12441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 02:30:25.666764 12316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } attempt: 5
I20260808 02:30:25.675628 12372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [term 1 FOLLOWER]: Refusing update from remote peer cca764302396443fb616ef6e1c1bd5aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Aug 08 02:30:25 dist-test-slave-pgkr krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786156224, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.231.61@KRBTEST.COM for kudu/127.19.231.60@KRBTEST.COM
I20260808 02:30:25.680361 12432 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 02:30:25.671864 12324 master_runner.cc:186] Error getting master registration for 127.19.231.61:46287: OK, instance_id { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" instance_seqno: 1786156224938459 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 02:30:25.685909 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.667391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.231.60:43725 (local address 127.0.0.1:42008)
0808 02:30:25.667625 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 02:30:25.667651 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 02:30:25.667795 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 02:30:25.669225 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 02:30:25.669230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 02:30:25.669813 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 02:30:25.670058 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.670073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.671661 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.671668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.673198 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 02:30:25.673217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.674535 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.674542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 02:30:25.674807 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.675519 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 02:30:25.675543 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 02:30:25.678420 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 02:30:25.682912 (+ 4492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.682917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.682921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.683450 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.683911 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 02:30:25.683916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 02:30:25.683920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 02:30:25.684097 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 02:30:25.684867 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 02:30:25.684872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 02:30:25.685040 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 02:30:25.685215 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 02:30:25.685526 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 02:30:25.686969 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:25.667803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.19.231.60:43725)
0808 02:30:25.668000 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.668018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:25.668039 (+ 21us) server_negotiation.cc:407] Connection header received
0808 02:30:25.668098 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:25.668103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:25.668217 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:25.668335 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:25.670248 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.671525 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.673402 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:25.674155 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:25.674300 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:25.678670 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 02:30:25.678677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 02:30:25.678682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 02:30:25.678728 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 02:30:25.682721 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.683599 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.683604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.683608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.683769 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 02:30:25.684232 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 02:30:25.684238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 02:30:25.684241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 02:30:25.684638 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 02:30:25.684794 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:25.686217 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0808 02:30:25.686461 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 02:30:25.686587 12324 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 02:30:25.693620 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba: Bootstrap starting.
W20260808 02:30:25.696233 12445 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 02:30:25.696411 12445 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 02:30:25.697774 12316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba: Neither blocks nor log segments found. Creating new log.
I20260808 02:30:25.701284 12316 log.cc:824] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba: Log is configured to *not* fsync() on all Append() calls
W20260808 02:30:25.717504 12347 tablet.cc:2369] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 02:30:25.720378 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba: No bootstrap required, opened a new log
I20260808 02:30:25.734627 12316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.735163 12316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20762a3ea21a4705b9890f034f5592ba, State: Initialized, Role: FOLLOWER
I20260808 02:30:25.735833 12316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } }
I20260808 02:30:25.741658 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cca764302396443fb616ef6e1c1bd5aa. Latest consensus state: current_term: 1 leader_uuid: "cca764302396443fb616ef6e1c1bd5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.742027 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:25.744580 12446 mvcc.cc:205] Tried to move back new op lower bound from 7316095900350578688 to 7316095900232568832. Current Snapshot: MvccSnapshot[applied={T|T < 7316095900350578688}]
I20260808 02:30:25.747557 12452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.748044 12452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:25.750411 12290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cca764302396443fb616ef6e1c1bd5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "20762a3ea21a4705b9890f034f5592ba"
I20260808 02:30:25.750383 12316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [sys.catalog]: configured and running, proceeding with master startup.
I20260808 02:30:25.751396 12409 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cca764302396443fb616ef6e1c1bd5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.751726 12409 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:25.753473 12441 catalog_manager.cc:1347] Generated new cluster ID: 7fb73bf35dbf4f1fb99143afc3672063
I20260808 02:30:25.753580 12441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 02:30:25.750911 12288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 02:30:25.759814 12288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [term 1 FOLLOWER]: Refusing update from remote peer cca764302396443fb616ef6e1c1bd5aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 02:30:25.763028 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cca764302396443fb616ef6e1c1bd5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.763265 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:25.764710 12290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca764302396443fb616ef6e1c1bd5aa in term 1.
I20260808 02:30:25.766790 12447 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 02:30:25.792232 12441 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 02:30:25.794133 12441 catalog_manager.cc:1504] Loading token signing keys...
I20260808 02:30:25.818646 12452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader cca764302396443fb616ef6e1c1bd5aa. Latest consensus state: current_term: 1 leader_uuid: "cca764302396443fb616ef6e1c1bd5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.818720 12409 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cca764302396443fb616ef6e1c1bd5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.819000 12452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:25.819047 12409 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa [sys.catalog]: This master's current role is: LEADER
I20260808 02:30:25.823482 12441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cca764302396443fb616ef6e1c1bd5aa: Generated new TSK 0
I20260808 02:30:25.825135 12441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 02:30:25.833189 12465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cca764302396443fb616ef6e1c1bd5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca764302396443fb616ef6e1c1bd5aa" member_type: VOTER last_known_addr { host: "127.19.231.62" port: 34189 } } peers { permanent_uuid: "20762a3ea21a4705b9890f034f5592ba" member_type: VOTER last_known_addr { host: "127.19.231.61" port: 46287 } } peers { permanent_uuid: "3232f92d73924500939f3c8a5ea9cb9c" member_type: VOTER last_known_addr { host: "127.19.231.60" port: 43725 } } }
I20260808 02:30:25.833933 12465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba [sys.catalog]: This master's current role is: FOLLOWER
I20260808 02:30:25.901821 12474 catalog_manager.cc:1259] Loaded cluster ID: 7fb73bf35dbf4f1fb99143afc3672063
I20260808 02:30:25.901948 12474 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba: loading cluster ID for follower catalog manager: success
I20260808 02:30:25.905757 12474 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba: acquiring CA information for follower catalog manager: success
I20260808 02:30:25.908325 12474 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20762a3ea21a4705b9890f034f5592ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 02:30:26.018707 12406 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:25.997183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.19.231.62:34189)
0808 02:30:25.997390 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:25.997397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.015794 (+ 18397us) server_negotiation.cc:407] Connection header received
0808 02:30:26.016840 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.016847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.016939 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.017022 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.018357 (+ 1335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45976: BlockingRecv error: recv got EOF from 127.0.0.1:45976 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 02:30:26.021174 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.001882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.19.231.60:43725)
0808 02:30:26.002089 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.002096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.019148 (+ 17052us) server_negotiation.cc:407] Connection header received
0808 02:30:26.019479 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.019485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.019574 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.019651 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.020859 (+ 1208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42018: BlockingRecv error: recv got EOF from 127.0.0.1:42018 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 02:30:26.021651 12321 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.013743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.19.231.61:46287)
0808 02:30:26.013983 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.013990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.019639 (+ 5649us) server_negotiation.cc:407] Connection header received
0808 02:30:26.020078 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.020087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.020179 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.020269 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.021403 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37998: BlockingRecv error: recv got EOF from 127.0.0.1:37998 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 02:30:26.072047 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.056363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.19.231.61:46287)
0808 02:30:26.056564 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.056604 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.063500 (+ 6896us) server_negotiation.cc:407] Connection header received
0808 02:30:26.064090 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.064098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.064332 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.064485 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.065229 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.066403 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.069029 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.069678 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.069768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.070516 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.071215 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.071333 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.071583 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.071688 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 02:30:26.113204 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.105962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.19.231.60:43725)
0808 02:30:26.106187 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.106193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.111174 (+ 4981us) server_negotiation.cc:407] Connection header received
0808 02:30:26.111489 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.111496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.111641 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.111727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.112948 (+ 1221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42034: BlockingRecv error: recv got EOF from 127.0.0.1:42034 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:30:26.162648 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.147367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.19.231.61:46287)
0808 02:30:26.147596 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.147604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.154677 (+ 7073us) server_negotiation.cc:407] Connection header received
0808 02:30:26.155271 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.155280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.155400 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.155477 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.156239 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.157418 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.159623 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.160399 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.160515 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.161686 (+ 1171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.161855 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.161922 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.162185 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.162275 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 02:30:26.208904 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.197427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.19.231.60:43725)
0808 02:30:26.197679 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.197688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.207230 (+ 9542us) server_negotiation.cc:407] Connection header received
0808 02:30:26.207492 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.207497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.207630 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.207737 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.208650 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42042: BlockingRecv error: recv got EOF from 127.0.0.1:42042 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 02:30:26.245663 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.235687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.19.231.61:46287)
0808 02:30:26.235880 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.235887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.239263 (+ 3376us) server_negotiation.cc:407] Connection header received
0808 02:30:26.239494 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.239500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.239590 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.239693 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.240260 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.241367 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.243489 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.244149 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.244287 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.244766 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.244925 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.245021 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.245212 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.245309 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 02:30:26.255169 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.252917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.19.231.60:43725)
0808 02:30:26.253171 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.253178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.253279 (+ 101us) server_negotiation.cc:407] Connection header received
0808 02:30:26.253620 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.253625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.253735 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.253853 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.254886 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42054: BlockingRecv error: recv got EOF from 127.0.0.1:42054 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 02:30:26.270259 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.263056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.19.231.61:46287)
0808 02:30:26.263323 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.263330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.263939 (+ 609us) server_negotiation.cc:407] Connection header received
0808 02:30:26.264131 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.264136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.264244 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.264353 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.264832 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.266036 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.268089 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.268821 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.268961 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.269452 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.269590 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.269702 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.269865 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.269947 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
W20260808 02:30:26.287753 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.284907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.19.231.60:43725)
0808 02:30:26.285244 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.285253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.286019 (+ 766us) server_negotiation.cc:407] Connection header received
0808 02:30:26.286285 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.286293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.286406 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.286523 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.287486 (+ 963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42070: BlockingRecv error: recv got EOF from 127.0.0.1:42070 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
I20260808 02:30:26.312731 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.304831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.19.231.61:46287)
0808 02:30:26.305031 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.305042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.305648 (+ 606us) server_negotiation.cc:407] Connection header received
0808 02:30:26.305891 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.305902 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.305988 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.306055 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.306728 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.307949 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.310254 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.311049 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.311210 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.311775 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.311970 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.312096 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.312282 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.312398 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 02:30:26.322397 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.320403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.19.231.60:43725)
0808 02:30:26.320619 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.320626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.320731 (+ 105us) server_negotiation.cc:407] Connection header received
0808 02:30:26.321012 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.321020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.321154 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.321299 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.322165 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42074: BlockingRecv error: recv got EOF from 127.0.0.1:42074 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:30:26.348238 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.341892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.19.231.61:46287)
0808 02:30:26.342122 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.342129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.342423 (+ 294us) server_negotiation.cc:407] Connection header received
0808 02:30:26.342672 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.342678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.342771 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.342897 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.343336 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.344606 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.345944 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.346652 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.346790 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.347444 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.347574 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.347658 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.347852 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.347935 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 02:30:26.358131 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.355538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.19.231.60:43725)
0808 02:30:26.355734 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.355740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.356315 (+ 575us) server_negotiation.cc:407] Connection header received
0808 02:30:26.356613 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.356619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.356719 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.356795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.357864 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42090: BlockingRecv error: recv got EOF from 127.0.0.1:42090 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 02:30:26.400810 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.394763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.19.231.61:46287)
0808 02:30:26.395047 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.395054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.395354 (+ 300us) server_negotiation.cc:407] Connection header received
0808 02:30:26.395557 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.395562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.395630 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.395712 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.396175 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.396898 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.398964 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.399451 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.399593 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.400027 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.400127 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.400225 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.400392 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.400458 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 02:30:26.409914 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.407756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.19.231.60:43725)
0808 02:30:26.407960 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.407966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.408497 (+ 531us) server_negotiation.cc:407] Connection header received
0808 02:30:26.408716 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.408720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.408795 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.408870 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.409677 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42104: BlockingRecv error: recv got EOF from 127.0.0.1:42104 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 02:30:26.482811 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.477592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.19.231.61:46287)
0808 02:30:26.477771 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.477776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.478209 (+ 433us) server_negotiation.cc:407] Connection header received
0808 02:30:26.478386 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.478391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.478460 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.478551 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.478937 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.479710 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.481222 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.481624 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.481718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.482143 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.482260 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.482326 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.482472 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.482536 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 02:30:26.492724 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.490153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.19.231.60:43725)
0808 02:30:26.490392 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.490401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.491010 (+ 609us) server_negotiation.cc:407] Connection header received
0808 02:30:26.491166 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.491171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.491280 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.491390 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.492331 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42106: BlockingRecv error: recv got EOF from 127.0.0.1:42106 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 02:30:26.629341 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.624164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.19.231.61:46287)
0808 02:30:26.624472 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.624480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.624666 (+ 186us) server_negotiation.cc:407] Connection header received
0808 02:30:26.624845 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.624849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.624922 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.625007 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.625457 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.626209 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.627587 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.628092 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.628194 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.628597 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.628721 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.628802 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.628949 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.629017 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260808 02:30:26.637280 12402 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 02:30:26.635355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.19.231.60:43725)
0808 02:30:26.635668 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.635675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.635900 (+ 225us) server_negotiation.cc:407] Connection header received
0808 02:30:26.636123 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.636127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.636208 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.636317 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 02:30:26.637100 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42110: BlockingRecv error: recv got EOF from 127.0.0.1:42110 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
I20260808 02:30:26.698840 12445 catalog_manager.cc:1259] Loaded cluster ID: 7fb73bf35dbf4f1fb99143afc3672063
I20260808 02:30:26.698949 12445 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: loading cluster ID for follower catalog manager: success
I20260808 02:30:26.701757 12445 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: acquiring CA information for follower catalog manager: success
I20260808 02:30:26.703718 12445 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3232f92d73924500939f3c8a5ea9cb9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 02:30:26.905175 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.900009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.19.231.61:46287)
0808 02:30:26.900233 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.900239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.900552 (+ 313us) server_negotiation.cc:407] Connection header received
0808 02:30:26.900728 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.900732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.900808 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.900909 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.901303 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.902192 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.903568 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.904002 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.904117 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.904537 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.904660 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.904741 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.904891 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.904957 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 02:30:26.918048 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.911683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.19.231.60:43725)
0808 02:30:26.911942 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.911949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.912201 (+ 252us) server_negotiation.cc:407] Connection header received
0808 02:30:26.912449 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.912456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.912655 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.912798 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.913248 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.914057 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.915575 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.916046 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.916213 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.916787 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.917361 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.917567 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.917627 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.917723 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 02:30:26.933305 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.924583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.19.231.61:46287)
0808 02:30:26.924842 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.924851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.927236 (+ 2385us) server_negotiation.cc:407] Connection header received
0808 02:30:26.927506 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.927512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.927621 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.927708 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.928255 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.929290 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.931341 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.931797 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.931935 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.932409 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.932528 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.932628 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.932853 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.932951 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:30:26.935453 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.924796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.19.231.60:43725)
0808 02:30:26.925028 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.925037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.928481 (+ 3444us) server_negotiation.cc:407] Connection header received
0808 02:30:26.928669 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.928674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.928741 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.928847 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.929663 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.930367 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.932714 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.933346 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.933477 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.934059 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.934191 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.934257 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.935006 (+ 749us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.935105 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 02:30:26.935873 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 02:30:26.924269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.19.231.62:34189)
0808 02:30:26.927066 (+ 2797us) server_negotiation.cc:206] Beginning negotiation
0808 02:30:26.927074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 02:30:26.927620 (+ 546us) server_negotiation.cc:407] Connection header received
0808 02:30:26.928390 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 02:30:26.928396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 02:30:26.928614 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 02:30:26.928799 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 02:30:26.929513 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.930682 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.932953 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 02:30:26.933762 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 02:30:26.933907 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 02:30:26.934590 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 02:30:26.935088 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 02:30:26.935219 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 02:30:26.935475 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 02:30:26.935569 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2314,"thread_start_us":2196,"threads_started":1}
I20260808 02:30:26.940624 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 12163
I20260808 02:30:26.941766 12234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:27.064952 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 12163
I20260808 02:30:27.094945 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 12246
I20260808 02:30:27.096338 12311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:27.207408 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 12246
I20260808 02:30:27.234818 20380 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu pid 12324
I20260808 02:30:27.236068 12393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 02:30:27.350987 20380 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOT6xt1/build/asan/bin/kudu with pid 12324
2026-08-08T02:30:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5070 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10152 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (387403 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST